Retrospect w8

There’s a rule of thumb for software development: make it, make it good, make it fast. For those unfamiliar with it, this means you should first build the core functionality of the software. Then, fix it’s bugs and make it as reliable as you can. Finally, optimize it to make it fast enough for your needs.

Retrospect 6 was a very good backup software, but its age was showing up. It was still fundamentally a Mac OS 9 app running on top of OS X Carbon API. The worst part about this was the need to launch the application in Finder, forcing you to have automatic login configured and remote-acess your backup machine via Remote Desktop or VNC, assuming it was running on a data center. Despite that, it was very reliable (I had two situations where Retrospect complained about corrupted data, and both situations were caused by faulty hardware). Also, it was not the fastest software I had seen, but it was good enough. Keep in mind Retrospect was designed when file systems had hundreds or a few thousand files on them, not a million or more like it’s normal today (I have about 1.5 million files on my laptop drive).

Continue reading

My walkman

Some of its features:

  • Unlimited songs!
  • No DRMs! Listen and record to whatever you want, legal or not! Just play it. Or record it!
  • Social features: Share your music! Plug two headphone sets, enjoy your favorite tunes with a friend, in stereo! Or switch tapes with your mates. No complicated wireless setup, no waiting for files to be transfered, no internet access required. The best social experience ever!
  • Built-in stereo microphone and line in. Records full quality sounds for as long as you want (with short breaks every 45 minutes)
  • Advanced space duplication algorithm: when you think the media is full, just warp it around and you have 100% more space!
  • No desktop, laptop, tablet, super-computer or any other damn thing with transistors in it required. No software. No platforms. No broadband internet connection. Nothing. It just works.
  • Strong, hard, heavy-duty case. Won’t break, period.
  • Very reliable, can work for decades, even under rough usage (trust me).
  • Replaceable built-in batteries. Can also be powered by external batteries or power supply.

Continue reading

WebObjects, PostgreSQL and DB growing, and growing, and growing…

One of the databases that support our application has some tables where data is updated very frequently. In PostgreSQL world, this means the table must be vacuumed very frequently to remove the old versions of the updated rows, clearing up space for new rows or new versions of existing rows. On very frequently updated tables, where insertions occur occasionally, but updates very often (several per second), recent versions of PostgreSQL should trigger their auto-vacuum daemon at a very high rate, like every 2 or 3 minutes, for those tables.

Despite this, our database kept growing like crazy, with no apparent reason. I decided to investigate what was going on.

The vacuum verbose is a neat command you can use in the PostgreSQL console to check what’s going on with the built-in cleaning service. This will run a vacuum on every table, giving you extensive report on how many used and unused rows exist in each table, and how many of the unused were deleted and their space reclaimed by PostgsreSQL.

The most interesting output of this command are the lines (one per table) like: DETAIL: 2942 dead row versions cannot be removed yet.. This means you have, in this example, 2942 rows on that table that are not used any more, but for some reason, vacuum was not able to delete them.

Browsing the web, I found out the most probable cause of this issue is the existence of a transaction that started a long time ago, and never finished. PostgreSQL will only delete rows older than the start timestamp of the oldest transaction still running. Newer versions might be needed by those transactions, so they are kept in place, even if their death sentence is already declared.

So, I started looking at what was going on inside the database engine, searching for those pending transactions. A lot of useful information about the current status of a PostgreSQL database can be found in many tables that exist in memory during PostgreSQL execution, that can be queried as any regular table. The one that was important for me was pg_stat_activity, which contains a list of all the current transactions.

By running the select * from pg_stat_activity; SQL command, I saw that each instance of my application had create two database connections, and for each instance, one of the connections had a transaction that started at the time the application instance was launched, but never committed or rolled back.

After some time of poking around, reading code, and talking to the right people, the problem was revealed. By default, WebObjects applications will open two connections to each database they talk to. One for the actual data queries, and the other to ask the database, at startup time, for what’s called the jdcb2info. This is a bunch of information needed to map between PostgreSQL and Java data types. And, due to one of those very old bugs that strangely nobody fixed yet that make WO fun, the transaction where that information is requested is never committed. So, the transaction will remain open, and PostgreSQL won’t vacuum a thing until you quit your application.

Although there’s not yet a solution for the problem itself (hopefully there will be, as I filled a bug in Radar), you can avoid it, by using the jdcb2info bundled in the PostgreSQL plugin instead of requesting it from the DB. To do this, simply append ?useBundledJdbcInfo=true to your DB connection URL, like jdbc:postgresql://localhost/dbname?useBundledJdbcInfo=true. This way, the second connection is not created at all, and no pending transaction will occur.

And of course, after doing this, the DB happily vacuumed hundreds of thousands of dead rows and stopped growing like crazy. Also, a tip: if you really need to vacuum the database right now, but you can’t stop your apps and interrupt service, simply obtain the pids of the PostgreSQL processes that are maintaing the offending connections from the pg_stat_activity table, and kill them, using the unix command kill <pid> command. The connections will die, but your apps will not.

WWDC 2009

Contrarily to WOWODC, I didn’t like WWDC at all this year. Some reasons for that were:

  • The IT track (the one that is of most interest for me) was very poor on this WWDC edition. There were a few talks about OS X Server and some of it’s new technologies, and that was it.
  • There was one (yes, one) talk about Java. This is ridiculous, specially if you consider that, a few years ago, Apple was selling the Mac as the best platform for developing and deploying Java applications. The first time I was at WWDC, 2006, there were a few Java sessions with interesting content. Don’t get me wrong, I love Cocoa, but some people need decent Java support to work.
  • No WebObjects content at all. The only official note about WebObjects was an announcement of an announcement that should happen within weeks. I don’t have any more details (and if I had, they would be under NDA, of course). This is sort of understandable, as they focused their attention on WOWODC, but it makes WWDC kind of pointless for people working on WebObjects projects in full time.
  • They somehow managed to serve even worse food than last year. Those guys must have some special talent, I thought it could not get worse than last years, buy they crossed that barrier at full speed. Also, there was no food on monday of tuesday evening, making it a little hard to eat and get back to watch the Apple Design Awards ceremony on time.
  • There were attendees distributing flyers and merchandising material inside the Moscone, despite the fact that doing that is clearly against the conference rules. Many people that attend WWDC work for some company, but a large part of the attendees own their own business, and everyone would love to promote it. But this is not a consumer exhibition, it’s a developer conference, with well defined rules, and people must respect them. What pissed me even more was the fact that the same flyers were distributed every day, by the same people, and no one from security kicked those guys out. It kinda looked like Portugal, where people break the laws with no apparent consequences.
  • Moscone center was at the limit of it’s capacity, and that showed. Although they could arrange the room sizes and session schedule way better than last year (I could attend all the sessions I wanted, which didn’t happen last year due to some of the rooms being full), there were not enough seats (either on the round tables, or the sofas) on the halls. On the first days it was really hard to find a place to rest for a while, or to write some lines of code.

I think Apple should seriously consider moving to a larger space, or breaking the WWDC in two separate conferences. Maybe one for newbies and another one for advanced users, or one for iPhone OS and another one for Mac and IT. I believe the WWDC is becoming less interesting every year, and the numbers back me up.

Apple claims that 60% of the attendees at WWDC 09 were first timers. Given that they sold out in 2008 and 09, and the Moscone capacity was the same on both events (around 5200 attendees), clearly a lot of people that came to 08 didn’t find it very interesting to come back this year. The fact that they sold out is based on the iPhone SDK hype, but people seem to be not coming back on the next year. I know I won’t, unless something deeply changes about WWDC.

I could not finish, of course, without congratulating João Pavão and the Sofa team for their well-deserved Apple Design Award on Versions, their SVN client. Really good job, guys! :)

WWDC 2009 (Keynote)

I found most of the keynote announcements interesting.

The hardware upgrades on the MacBook Pros were cool. First, Apple seemed to acknowledge that the difference between the unibody MacBook and the MacBook Pro was not strong enough, so the MacBook is dead, long live MacBook Pro. This means the 13 inch laptop has now some of the Pro features, namely Firewire was brought back, in it’s 800 Mbps version. Other nice stuff, like the non-removable high capacity battery and the back-lighted keyboard made their way to the MacBook Pro 13”. The usual CPU speed bumps were there, of course. The only decision that left people a bit confused was the introduction of the SD-card reader. Not only it seems useless (because you may plug any SD-card enabled device to your Mac using USB) but because it kicked out some old features on the Macs that got it. The Express card slot went away on the MacBook 15”, and the Audio in port on the MacBook (now Pro) 13” was replaced by an Audio in/out jack found on the iPhones. Although you can buy an Apple headset to plug there, there’s an obvious question not yet answered, how to record audio from the line in?

The MacBook Air also got some CPU upgrades, and very interesting price cuts. Finally they are not ridiculously overcharging (at least, not as much as they did) the MacBook Air form factor, despite the fact that the machine specs were disappointing. I think they should ship them with 4 GB of RAM instead of 2 (specially because it’s not upgradable), but despite that it’s becoming a really interesting machine to buy if you travel a lot. The only feature I miss (and I was hoping to find it on this new machines) is a 3G modem, which would avoid carrying one more USB dongle. But they may be because of…

… the iPhone. Apple surprised me with the new iPhone 3GS and the 3.0 version of the OS. Despite the ugly name, the 3GS is a really nice improvement to the old model, and the differences are in the details. Among all, we have:

  • (3GS only) Hardware encryption of all your data. That’s really cool on a device that can easily be lost or stolen. This helps keeping your data private, but it’s not all.
  • Find my iPhone. Every mobile phone user went trough this at least once. You look for your iPhone, but it seems to be nowhere. If it’s somewhere around your house, you just grab the landline phone, call it, and follow the ringtone. But sometimes, no ringtone will sound, and that’s the time you start panicking. Did you left it in the car? Or maybe in a restaurant? Or, worse, someone stole it without you even noticing? Now, if you subscribe MobileMe, you can simply access the service, and ask for the current localization of the phone. If it’s turned on, the iPhone will report it’s current position (based on the usual localization facilites, like GPS or WiFi triangulation) that will be presented on a map. But there’s more: you may ask the phone to make some noise, and display a message with your landline phone number and address, and with luck, someone will pick it up and return it to you, or deliver it to a local police station. There’s the nice detail that the phone will make noise even if it’s set to silent. But, even better, if your phone is lost forever, you can still keep your data private. Just beam it an order to wipe all the data on it, and the phone will start erasing your stuff. If it’s encrypted, the wipe is immediate. Later, if you get back your iphone, just plug it to your Mac, and the content will be restored from your backups (which can also be encrypted, for extra protection).
  • Tethering. This one is big, at least for me. iPhone will share it’s 3G connection either trough the USB cable or Bluetooth. This is awesome, allowing you to pay only one 3G subscription instead of two (one for the phone, and one for the laptop). Also, no more crappy 3G modems with questionable quality drivers.

There are also some nice details, like the faster CPU on the 3GS, MMS support, better camera, etc. The iPhone is becoming more and more interesting, and I may find myself finally getting one, if the portuguese operators manage to sell them for decent (or not very indecent) prices.

FireFox 3: the evaluation order bug is back

A few months ago, I wrote about a Javascript evaluation order bug I hit on Firefox 3. I managed to create a workaround and everything was going fine. I assumed the bug had been fixed on the subsequent minor releases of Firefox.

Some time ago, that same page started to be the target of some bug reports stating that users would occasionally get a blank page when accessing it, and the only way to get the page back was to hit reload. This was happening on Firefox 3.0.7 and newer.

As I wrote on the previous post, our page does an RPC call while loading the header scripts, like this:

<script src="/some/path/prototype.js"></script>
<script src="/some/path/effects.js"></script>
<script src="/some/path/wonder.js"></script>
<script src="/some/path/english.js"></script>
<script src="/some/path/combo.js"></script>
<script src="/some/path/feedback.js"></script>
<script src="/some/path/jsonrpc.js"></script>
<script type="text/javascript">
var jason = new JSONRpcClient("proxyUrl");
</script>
<script src="/some/path/comboboxes.js"></script>
...

After poking around with Firebug, I discovered that all the code below the call was being ignored by the Firefox parser. That would naturally result in an empty body, which lead to a blank page.

I started googling around, and I found two interesting Firefox bug reports, 444322 and 478277.

The first of those bugs mentions the original Javascript evaluating order issue I wrote about before (which was not, in fact, a JavaScript evaluating order, but an issue where the parser would not wait for the result of the RPC call to continue parsing). They also mention the Firefox team released a “fix” for it in 3.0.6, but people kept reporting the issue was not yet fixed (some claimed it in fact got worse), which lead to opening the second bug report. Also, an important fact about this bug is that it seems to happen only when all the page resources are already cached locally by the browser. This includes all the resources loaded in the page header (usually, JavaScript and CSS files).

A few interesting comments (#30 and #34) clarify what’s causing this, and mentions this is a piece of “fragile code”. I don’t know enough about the Firefox code base (which is a nice way to say I know nothing at all) to be sure about this, but I believe the “fix” caused an even bigger problem, where, in some conditions, the parser will enter a state where it will simply eat all the input without parsing it. This leads to the blank page problem.

Well, this is all very interesting, but I had a problem that needed to be solved. So, I hacked. On of the things that avoid this bug to be triggered is at least one of the page resources to not exist on the local cache when the page is loaded. So, I forced this situation to happen. I picked up a small JS file that is loaded before the Ajax call, and configured Apache to add all the necessary headers for browsers to not cache it. This is done with something like this:

<Files "somefile.js">
	Header set Cache-Control: "private, pre-check=0,
post-check=0, max-age=0"
	Header set Expires: 0
	Header set Pragma: no-cache
</Files>

This will prevent browsers from caching the file. Remember to to this on a small file, or you’ll be wasting bandwidth and causing the page load time to increase.

Now if only someone would really fix that bug… :)

WebObjects, SSL, Apache 2.2, FreeBSD problems

I still didn’t have time to investigate this deeply, but here goes anyway, as I didn’t find the answer to this anywhere.

I had the problem described in the Apache 2.2 adaptor: SSL fails on FreeBSD amd64 thread in the WebObjects-Deploy mailing list. As the thread suggests, the problem is not related to 64 bits at all, but with a problem related to the SSL client certificate becoming corrupted somewhere between an SSL request entering apache and leaving to the selected WebObjects application instance.

The thread suggests a little hack in the Wonder’s Adaptor code to ignore the SSL client certificate stuff, but I found that to be a little too dirty. So I dug this as much as time allowed.

I found the adaptor will forward all the SSL http headers it has to the application. This allows the application to analyse the SSL information (like the certificates themselves, the certificate information, etc) if, for some reason, that’s needed. However, the SSL client certificate information is becoming corrupted, and screws up the request, making it invalid. As the application will not reply to what it believes to be an invalid http request, the adaptor thinks the application is non-responsive, and considers it invalid for some minutes. That’s why the user starts to get the infamous “No instance available” message.

I still don’t know if this is an Apache, mod_ssl, WO Adaptor or whatever bug. But I know how to work around it, avoiding this error. As the problem is related to providing the SSL information to the adaptor, I chose to simply block it. This can be done in the Apache configuration. Just add this to the right context (usually the SSL virtual host configuration block):


SSLOptions -FakeBasicAuth -ExportCertData -StrictRequire -StdEnvVars

This will hide all the SSL information from the adaptor, avoiding the error. I’ll try to dig this a little more if I have some free time and update this post.

Apple 2009 wish list

It’s a brand new year. So here’s my wish list for Apple:

  • Please fix the wireless driver that causes my Mac to crash about 10% of the times I turn Airport off.
  • Please fix the trackpad driver, or whatever is causing the trackpad to behave strongly erratic during about 30 seconds after waking the Mac up.
  • Please fix the damn copy/paste bug that makes the paste command paste the previously copied object and not the most recent one. This is specially irritating when you cut a piece of text, paste and you realize you are pasting something else, and that your supposedly cut piece of text is lost forever, unless you can undo and get it back.
  • Please fix the irritating bug that causes an iChat window to keep being the active one even after I click Safari, making its window go in front of iChat’s. That’s specially annoying when I type apple-W to close the Safari window, and the ichat one goes away.
  • Please provide replacement keyboards for people who has pre-unibody MacBook Pros that, you know, actually sense a keystroke every time the key goes all the way down, without the need to almost punch the key.
  • Please fix whatever is causing my father’s MacBook Pro to keep waking up and going back to sleep when the lid is closed and the charger on, despite I had already turned off every god damn thing that could wake it up, including the lid open event.
  • Speaking about the charger, please provide chargers where the charge light doesn’t go off for some unknown reason. It still works, but it doesn’t inspire a lot of confidence in it ans it’s safety.
  • Please provide granular updates to Mac OS X Server. Please please please pretty please.
  • Please care a little more about the entreprise and IT markets, namely your own web application technology (WebObjects, of course).

Thank you, guys! You must hate me but you’re nice people anyway. Sometimes.