RESTNotes code sample

A few days ago I released RESTNotes. RESTNotes is a code sample package of a distributed notes application. It consists of an iPhone application that stores all its data in an WebObjects server app. Communication is done using REST style techniques.

I’ve done this to learn about REST, and get the feeling of how should things be made, what problems would I find, and so on. I spent some time creating an easy-to-use methodology on the iPhone side, making the communication APIs simple but flexible. Also, I looked at problems like authentication and error reporting.

If you’re interested, the source is published on Google Code, together with some nice documentation on the project wiki, explaining the general architecture of the client and solutions to some of the problems I found. Hope you find it useful. :)

WebObjects, PostgreSQL and DB growing, and growing, and growing…

One of the databases that support our application has some tables where data is updated very frequently. In PostgreSQL world, this means the table must be vacuumed very frequently to remove the old versions of the updated rows, clearing up space for new rows or new versions of existing rows. On very frequently updated tables, where insertions occur occasionally, but updates very often (several per second), recent versions of PostgreSQL should trigger their auto-vacuum daemon at a very high rate, like every 2 or 3 minutes, for those tables.

Despite this, our database kept growing like crazy, with no apparent reason. I decided to investigate what was going on.

The vacuum verbose is a neat command you can use in the PostgreSQL console to check what’s going on with the built-in cleaning service. This will run a vacuum on every table, giving you extensive report on how many used and unused rows exist in each table, and how many of the unused were deleted and their space reclaimed by PostgsreSQL.

The most interesting output of this command are the lines (one per table) like: DETAIL: 2942 dead row versions cannot be removed yet.. This means you have, in this example, 2942 rows on that table that are not used any more, but for some reason, vacuum was not able to delete them.

Browsing the web, I found out the most probable cause of this issue is the existence of a transaction that started a long time ago, and never finished. PostgreSQL will only delete rows older than the start timestamp of the oldest transaction still running. Newer versions might be needed by those transactions, so they are kept in place, even if their death sentence is already declared.

So, I started looking at what was going on inside the database engine, searching for those pending transactions. A lot of useful information about the current status of a PostgreSQL database can be found in many tables that exist in memory during PostgreSQL execution, that can be queried as any regular table. The one that was important for me was pg_stat_activity, which contains a list of all the current transactions.

By running the select * from pg_stat_activity; SQL command, I saw that each instance of my application had create two database connections, and for each instance, one of the connections had a transaction that started at the time the application instance was launched, but never committed or rolled back.

After some time of poking around, reading code, and talking to the right people, the problem was revealed. By default, WebObjects applications will open two connections to each database they talk to. One for the actual data queries, and the other to ask the database, at startup time, for what’s called the jdcb2info. This is a bunch of information needed to map between PostgreSQL and Java data types. And, due to one of those very old bugs that strangely nobody fixed yet that make WO fun, the transaction where that information is requested is never committed. So, the transaction will remain open, and PostgreSQL won’t vacuum a thing until you quit your application.

Although there’s not yet a solution for the problem itself (hopefully there will be, as I filled a bug in Radar), you can avoid it, by using the jdcb2info bundled in the PostgreSQL plugin instead of requesting it from the DB. To do this, simply append ?useBundledJdbcInfo=true to your DB connection URL, like jdbc:postgresql://localhost/dbname?useBundledJdbcInfo=true. This way, the second connection is not created at all, and no pending transaction will occur.

And of course, after doing this, the DB happily vacuumed hundreds of thousands of dead rows and stopped growing like crazy. Also, a tip: if you really need to vacuum the database right now, but you can’t stop your apps and interrupt service, simply obtain the pids of the PostgreSQL processes that are maintaing the offending connections from the pg_stat_activity table, and kill them, using the unix command kill <pid> command. The connections will die, but your apps will not.

WebObjects, SSL, Apache 2.2, FreeBSD problems

I still didn’t have time to investigate this deeply, but here goes anyway, as I didn’t find the answer to this anywhere.

I had the problem described in the Apache 2.2 adaptor: SSL fails on FreeBSD amd64 thread in the WebObjects-Deploy mailing list. As the thread suggests, the problem is not related to 64 bits at all, but with a problem related to the SSL client certificate becoming corrupted somewhere between an SSL request entering apache and leaving to the selected WebObjects application instance.

The thread suggests a little hack in the Wonder’s Adaptor code to ignore the SSL client certificate stuff, but I found that to be a little too dirty. So I dug this as much as time allowed.

I found the adaptor will forward all the SSL http headers it has to the application. This allows the application to analyse the SSL information (like the certificates themselves, the certificate information, etc) if, for some reason, that’s needed. However, the SSL client certificate information is becoming corrupted, and screws up the request, making it invalid. As the application will not reply to what it believes to be an invalid http request, the adaptor thinks the application is non-responsive, and considers it invalid for some minutes. That’s why the user starts to get the infamous “No instance available” message.

I still don’t know if this is an Apache, mod_ssl, WO Adaptor or whatever bug. But I know how to work around it, avoiding this error. As the problem is related to providing the SSL information to the adaptor, I chose to simply block it. This can be done in the Apache configuration. Just add this to the right context (usually the SSL virtual host configuration block):

SSLOptions -FakeBasicAuth -ExportCertData -StrictRequire -StdEnvVars

This will hide all the SSL information from the adaptor, avoiding the error. I’ll try to dig this a little more if I have some free time and update this post.

Apple 2009 wish list

It’s a brand new year. So here’s my wish list for Apple:

  • Please fix the wireless driver that causes my Mac to crash about 10% of the times I turn Airport off.
  • Please fix the trackpad driver, or whatever is causing the trackpad to behave strongly erratic during about 30 seconds after waking the Mac up.
  • Please fix the damn copy/paste bug that makes the paste command paste the previously copied object and not the most recent one. This is specially irritating when you cut a piece of text, paste and you realize you are pasting something else, and that your supposedly cut piece of text is lost forever, unless you can undo and get it back.
  • Please fix the irritating bug that causes an iChat window to keep being the active one even after I click Safari, making its window go in front of iChat’s. That’s specially annoying when I type apple-W to close the Safari window, and the ichat one goes away.
  • Please provide replacement keyboards for people who has pre-unibody MacBook Pros that, you know, actually sense a keystroke every time the key goes all the way down, without the need to almost punch the key.
  • Please fix whatever is causing my father’s MacBook Pro to keep waking up and going back to sleep when the lid is closed and the charger on, despite I had already turned off every god damn thing that could wake it up, including the lid open event.
  • Speaking about the charger, please provide chargers where the charge light doesn’t go off for some unknown reason. It still works, but it doesn’t inspire a lot of confidence in it ans it’s safety.
  • Please provide granular updates to Mac OS X Server. Please please please pretty please.
  • Please care a little more about the entreprise and IT markets, namely your own web application technology (WebObjects, of course).

Thank you, guys! You must hate me but you’re nice people anyway. Sometimes.

Firefox 3 javascript evaluating order problems

I found out the hard way the recently released third main version of Firefox has a serious issue related to the order in which the Javascript code is executed in a page. The bug is somehow related to the files being, or not, cached on the browser.

Wonder Ajax framework adds a lot of Ajax goodies to a standard WebObjects applications. The way a page is built is by placing on the page header all the calls to load Javascript files. There’s a special case, which is when you need a JSON proxy. That is useful if you need to write more powerful Ajax behaviour than all the component trickery Wonder offers you. The JSON proxy will not only load the necessary Javascript file, but run one code line to create the actual proxy and establish the communication with the server. So the page header will have something that looks more or less like this:

<script src="/some/path/prototype.js"></script>
<script src="/some/path/effects.js"></script>
<script src="/some/path/wonder.js"></script>
<script src="/some/path/english.js"></script>
<script src="/some/path/combo.js"></script>
<script src="/some/path/feedback.js"></script>
<script src="/some/path/jsonrpc.js"></script>
<script type="text/javascript">
var jason = new JSONRpcClient("proxyUrl");
<script src="/some/path/comboboxes.js"></script>

When the page loads, all those scripts will be loaded and the new JSONRpcClient() method will be executed during the page load, as expected.

The problem starts when you have a javascript method associated with the onLoad window event. This is done by using the onload attribute in the body element:

<body onLoad="initPage();">

The onLoad function should be triggered when the page finishes loading. This, implicitly, means that the onLoad will be run only after all the inline javascript code (in the page header or body) be run, because that is still considered to be part of the page loading process. So, in our case, first, load all the JS and create the JSON proxy, and then run the onLoad method.

This works on every browser, and also on Firefox 3 on the first page load. The problem is that most of the subsequent page loads on Firefox 3 won’t work, at least if your onLoad method depends on the existence of the JSON proxy. For some reason, when Firefox 3 has all the Javascript files already in the local cache, the onLoad event is triggered too soon, namely before the header scripts are executed and the proxy has been created. This will break all your code executed by the onLoad method that assumes the proxy is already ready to work.

I have tried several solutions for this. The only one I had success with is the following. It’s kind of dirty, but it works. The idea is simple: if the onLoad method is to be executed before the proxy exists, than we wait some miliseconds and try again later until the proxy is ready to be used. So, if your method is like this:

function myOnloadMethod() {

Change it to:

function myOnloadMethod() {
  if( window.jason ) {
  } else {
    setTimeout( 'myOnloadMethod();', 10 );

This will check every 10 milliseconds if the proxy is already there. If it is, your method will be run. If not, it will keep trying.

Upgrading Mac OS X Leopard Server

I was expecting that. Fortunately, I was ready for it. So, I’ll describe some details of upgrading Leopard Server from 10.5.2 to 10.5.4 to avoid you the same trouble.

If you read my previous article about Leopard Server, you know that I had to recompile MySQL, PHP and Apache to PowerPC 32 bits, because I could not use the 64 bits version on our G5 server. The story was a bit complicated, but the idea was that I needed PostgreSQL and MySQL support on PHP. As the MySQL default installation on OS X Server doesn’t include the shared libraries, I had to recompile MySQL to be able to recompile PHP. But I could not make MySQL compile for 64 bits, so I compiled everything (including Apache) to 32 bits.

So, I was expecting it: as soon as I would upgrade my server, the installers would replace Apache with a more recent version, and it would not start because PHP and MySQL stuff was compiled for 32 bits. And it happened.

How to fix it? Simple. You have to do two things: first, remove the PowerPC 64 part from the httpd binary. Second, recompile your customized version of PHP, because PHP gets replaced too.

To remove the PowerPC 64 part of the binary, you don’t need to recompile Apache as I did before. There are two very useful commands you can use: file and lipo. If you do the following commands:

cd /usr/sbin
file httpd

The result will be:

httpd: Mach-O universal binary with 4 architectures
httpd (for architecture ppc7400): Mach-O executable ppc
httpd (for architecture ppc64): Mach-O 64-bit executable ppc64
httpd (for architecture i386): Mach-O executable i386
httpd (for architecture x86_64): Mach-O 64-bit executable x86_64

As you can see, Apple ships httpd compiled for their four possible architectures. In my case, I want to remove the ppc64 one to force httpd being executed in 32 bits. That’s where lipo comes in hand:

lipo -remove ppc64 -output httpd32 httpd
mv httpd httpd64
mv httpd32 httpd

That’s it. The ppc64 part of the binary has been removed, and a new binary was created. I then just switch the binaries, and there you go, Apache running on 32 bits mode on PowerPC:

httpd32: Mach-O universal binary with 3 architectures
httpd32 (for architecture ppc7400): Mach-O executable ppc
httpd32 (for architecture i386): Mach-O executable i386
httpd32 (for architecture x86_64): Mach-O 64-bit executable x86_64

Then I just needed to recompile PHP and everything was back on track.

Also, be careful because the 10.5.4 update will upgrade WebObjects to 5.4.2. If you are still using WO 5.3 like me, you’ll have to reinstall 5.3 again, as described in the Mike’s document.

Back from USA

Well, I’m back from another WOWODC and WWDC. I’m still really tired, but some quick notes:

  • As David LeBer already mentioned, Pascal did an amazing job organizing WOWODC all by himself. Great room (a bit cold on the first day ;) ), large windows and sunlight on the halls. Food (well… not that good, but after all, it’s USA!) and caffeine provided frequently. Very very nice. Suggestion for the next year: a bigger (and brighter) screen, and eventually plasma screens among the room to make it easy for people in the back to read the code.
  • I learned a lot about WO frameworks out there (like Wonder, Houdah and specially LEWOStuff that I did not know before). I met for the first time some very talented people, and of course, all the folks from the previous conferences. It’s great to be able to have technical discussions and know different views on the same problems from all those skilled and experienced people out there, face to face.
  • WWDC had some interesting news on many stuff. As you know, I cannot talk about the stuff under NDA, so I shall only say that some interesting stuff is being done on the WO side. Also, as you all know by now, the iPhone is now 3G, includes a GPS, the price was slashed, and will be available in many countries of the world. I just hope the service providers slash the data roaming prices, because that makes the iPhone useless when you go to foreign countries. Finally, Snow Leopard was announced, and, as already expected, the focus is not on new features, but on a big cleanup of the OS infrastructure. Not only this are great news for us, developers, but also shows some courage from Apple and a lot of respect for their users. They want to focus the next year on improving the quality of their OS, rather than packing it up with some new features just to win the race against the competition.
  • As a side note, the MacTech people was giving away some magazines for free to the people who were standing in line during the morning. I took the time to read most of it during my flight, and I really liked it. I was a MacTech subscriber in the past, but I cancelled it because, during my graduation, I didn’t have time to read it (it’s good to graduate on a place where you actually don’t have time to learn, isn’t it?). Maybe I’ll subscribe it again now.
  • Not related to the conferences themselves, we went to visit the bay area surroundings on Friday afternoon. We did the classic trip to the Apple and Google campuses, because we are all geeks, but we also went to the Stanford and Berkeley campuses. The Standford campus totally blow me away. You have to see it to believe it. From now on, I’ll laugh, really laugh, every time I hear a faculty from my university stating that we actually have a campus. The Berkeley campus did not impress me much. It’s more urban style, more crowded and dense. I prefer the Stanford way, with space, a huge amount of space, tons and tons of space, really. Almost made me want to return to the univ! :)