Terminal.app

In 1997. (Yes, I’m cleaning up some really old stuff)

or How I learned to stop worrying and lock the EOObjectStoreCoordinator

So those of you who read the WebObjects Developer mailing list know I’m a bit obsessed about concurrency issues. Namely concurrent data write operations, specially when you have lots of derived data on the database due to performance issues.

A few months ago, I wrote an article about handling concurrent data access in WebObjects applications that included a nice solution for the problem. The thing is… it’s wrong. It’s all wrong. It doesn’t work, in the sense that it won’t give you any guarantee that you won’t discard upgrades made by other threads or instances of the application. I have to thank Chuck that looked at the code and said “doesn’t work”. Chuck is right, by definition, so when he says something doesn’t work, it just doesn’t. And he was right again.

So, in that article, I avoided the problem created by an editing context delaying the merge of the changes made by other contexts to its objects to the moment when the context is unlocked. But I made a very important mistake: I assumed WebObjects implementation of optimistic locking worked. It doesn’t. Unfortunately, optimistic locking is broken in WO due to a design flaw.

The problem is simple: you can lock everything you want. You can lock your contexts, you can use classic Java locking, you can do whatever you want. But, as long as you don’t lock the EOObjectStoreCoordinator, you can’t trust optimistic locking. Because the row snapshots are unique to a EOF stack, you can have a thread in the middle of a critical part of your code, like I showed on the previous article, and have another thread changing those snapshots simply by doing any operation that brings fresh data from the database, like a fetch specification with the refreshesRefetchedObjects option turned on. Unless you use classic Java locking to lock everything in your code where you might eventually do a refreshing fetch (which is insane), you cannot trust the row snapshots that will be used as a base of comparison for optimistic locking to be unchanged during your code critical segments.

If this sounds too confusing, think in the following example. We have an entity with attribute ATT that we want to increment using optimistic locking. Remember that the ideia is the following: we want to make sure that, if two or more threads increment the value at the same time, only one of the threads will succeed, all the others will receive an optimistic locking exception, and effectively abort the transaction (or repeat it).

So, we have two instances, INST1 and INST2 running. Assume we increment the value inside a method like the one I wrote in the previous article. Let’s now run some steps:

1. INST2 reads ATT with value = 3. The row snapshot in the INST2 EOF stack has ATT = 3, as expected.
2. INST1 reads ATT with value = 3. Again, the row snapshot in INST1 EOF stack has ATT = 3.
3. INST2 increments ATT to 4, and saves the result to the database. It will naturally succeed, updating the DB and it’s own snapshot to ATT = 4. We can forget INST2 from now on.
4. INST1, in another thread, runs a fetch specification with refreshesRefetchedObjects on. As the ATT value in the database is now 4, that thread will have ATT = 4 on the editing context that fetched the object, and (here’s the problem) the row snapshot will be updated to what’s currently in the database, which is ATT = 4.
5. INST1, in the original thread, finally increments the value of ATT to 4 (remember, it was 3 on it’s editing context), and tries to save. It will use the row snapshot do to the optimistic locking comparison. So, what it will do, in a rough translation of SQL, is “update att to 4 where entityId is some Id and att is 4“. The red 4 comes from the row snapshot.
6. As the value on the database is now 4, the save will succeed. The final value of ATT will be 4 and not 5, as expected after being incremented in both threads.

The problem here is the assumption EOF does when saving changes of attributes with optimistic locking enabled. EOF assumes that the row snapshots represent the original status of the objects when they were fetched to the context being saved. So, it compares the current database status with the original status to detect changes. The problem is that assumption is simply not true. There can only be one row snapshot in an EOF stack for a given row, and nothing will guarantee the snapshot doesn’t change in the middle of the critical section.

Facing this, there’s only one thing you can do: make that assumption become true. And to do that, you need to either use a different EOF stack per thread (very very high resource consumption, not only in memory but in DB connections too) or use the “big” lock in the stack. That big lock is the lock in the EOObjectStoreCoordinator. If you need absolute reliability when managing concurrent data access, you’ll have to lock EOObjectStoreCoordinator before fetching (and refreshing, of course) the objects you will change and only unlock it after saving those objects. You can do that with code like this:

EOObjectStore osc = someEditingContextOnThisThread.rootObjectStore();
try {
  osc.lock();
  // Do as I wrote on the previous article:
  // create a new editing context with fetch timestamp
  // set to "now", fetch all your objects, process them,
  // and try to save them, handling OL exceptions.
} finally {
  osc.unlock();
}

Note that I put osc.unlock() call in the “finally” segment of the try/catch wrapper. You need to take extra care to make sure that, whatever happens, you will not leave the EOObjectStoreCoordinator locked, or all your application’s database access will simply be stopped forever.

Of course, there’s a downside for this solution: while you are processing the objects inside the critical code segment, no other thread will be able to access the database, in read or write mode. I can give you two advices on this.

First, the obvious one: keep that processing as simple and fast as possible. Remember that long operations should be done in background threads with their own EOF stack or, better yet, in separate “maintenance” applications. This technique should be used only for fast operations, or the users of your application will get bored waiting.

Second, create multiple EOF stacks (a reasonable amount, like 3 or 4). This will speed up things a bit, because it will reduce the probability of a thread having to wait for another one. If you have 4 EOF stacks and you enter a critical part of your code on one of them, 75% of your threads (and that means 75% of your users) won’t take the hit. The downside is more memory usage. You’ll have multiple copies of the row snapshots, plus all the objects needed to handle the database connection. And, of course, you may have to increase the number of maximum simultaneous connections on the database, which may force you to increase the maximum shared memory and semaphores on your operating system configuration. The good part is that, if you are using Wonder, creating the EOF stacks themselves is as simple as adding a property to your Properties file. That’s right! Just set er.extensions.ERXObjectStoreCoordinatorPool.maxCoordinators=x, being x the number of EOF stacks you want. That’s it, really. It just works, and it tries to minimize weirdness by creating all the editing contexts on a session in the same stack. It’s that good. It’s Wonder!

I finally moved to an Intel machine. Despite the dramatic speed improvement in everything Java-related, namely Eclipse, there’s another big advantage: being able to run IE on Windows using a virtual machine. Unfortunately, that’s something every web developer must do to ensure his or her application will work on the most used (and crappy) browser on earth.

I installed Parallels and created two virtual machines, one for IE 6 and another one for IE 7. This way I’m sure there are no weird problems between those two versions (having more than an IE version on Windows can only be accomplished by hacks, and hacks are bad). Also I can install Visual Web Developer Express Edition on each of the VMs, and use either IE 6 or 7 to debug.

My apps run inside virtual hosts on Mac OS X apache, under a fake DNS name. On Mac OS X it’s easy to add the DNS entry to the /etc/hosts file, under the 127.0.0.1 entry. This way, your DNS name will always point to your mac, and you’ll be able to reach your virtual host.

I wanted to do the same from inside Windows running on Parallels. An easy way would be to edit the Windows hosts file, adding the Mac OS X public IP to the file. But that will only work if the OS X IP doesn’t change. My Intel mac is an MBP, and I change the network I use often, so I needed a little more flexibility. So, this is the way I found to do this:

1. Configure your VM to use Shared Networking. This wall, Parallels extensions installed on your Mac will create a NAT network where your virtual machine will be hooked into.
   
   
2. Open Mac System Preferences, and look for the “Parallels NAT” network port. This is an interesting one, because it allows the Mac itself to be connected to the virtual NAT network, using an IP on the NAT subnet. Write down that IP: this will be the IP you’ll use to access the Mac virtual hosts from within the virtual machines.
   
   
3. Finally, edit Windows hosts file. This file is located on \WINDOWS\system32\drivers\etc\hosts. Add a line with the IP (in my case, 10.211.55.2) and the name of the virtual host, just like you do on the Mac.
   
   

That’s it. Now you can access your Mac virtual hosts from Windows, whatever the Mac IP is. Ick, what’s a Windows screenshot doing in my blog!?

This was it. I spent the easter weekend migrating GAEL’s Xserve to Leopard Server. It all went well, although some more or less serious issues poped up.

Our server is used mainly for hosting web content and applications (php, perl, and of course, WebObjects). It also handles some Subversion repositories and some other minor stuff.

I did the standard procedure I always do when migrating a machine to a new OS version: clone the hard drive to an external firewire disk, format the internal hard drive, install new OS and migrate data. I usually use migration assistant, but of course, this is a server, it’s a little more complicated than that.

While my memory is still fresh, here’s some notes about it, not necessarily by any specific order.

RAID Formatting

Our Xserve has two 80 GB hard drives configured in software RAID 1 (mirroring). As Apple sometimes does some tweaks and changes on the RAID software and drivers, I decided to destroy the RAID and create a new one. So I did: I booted from the Leopard Server DVD, destroyed the RAID and tried to make a new one. I had some problems with that, though. Disk Utility was not allowing me to create the new RAID. I don’t recall the error exactly, but it had something to do with not being able to mount a volume or RAID slice. I quit Disk Utility and launched it again, and then the RAID creation went fine. I just love to see those two blue leds blinking in sync!

Installation and System Updates

Installation itself went without any issues. Our Xserve has a graphics card, so it was like any regular desktop Mac, click click choose click and wait. The system installed correctly, rebooted, configuration assistants, answered all the questions, network working, etc. Perfect. Then, I went to grab all the system updates. I installed it this weekend, so I had a few updates waiting, namely the 10.5.2 combo update. Then something weird happened - after installing all the available updates, the machine rebooted 3 times in a row instead of just one. I know some updates that came out lately require 2 reboots in a row, but I never had seen 3. When the server finally came to life, I manually rebooted it again 2 or 3 times more, just to see if it was booting OK. Apparently, everything is fine. I checked the logs, and they were inconclusive. So, does anyone know if 3 reboots in a row is normal for all the updates that came out so far for a G5 Xserve running Leopard?

SSH

This is a fast one, but… sshd comes with PermitRootLogin defaulting to “yes”. Oh come on, guys!

User Migration

This is one of the most serious issues that I find with Mac OS X Server migration. I had seen this when migrating from Panther Server to Tiger Server, and it’s still a problem. The thing is: you cannot migrate passwords. You can use Workgroup Manager to export all the user information… except passwords. That means all the user passwords will have to be reset on the new server. Of course, I don’t expect the real passwords to be exported - specially because they are hashed, so it’s impossible to recover them. But the hash itself could be exported and imported again.

This presents a very serious issue to system administrators and users. Of course, if you have thousands of users, you should use multiple LDAP servers dedicated to the authentication services, and you can clone them at will, making sure that you never loose information and the service never stops. But when you have about 30 users like we do, that is overkill. Even so, it’s a real pain in the ass to reset all those passwords, because some users are actually not in our office. They are external users, either from the other university campus (although that’s not too bad, I actually live closer to that campus that the one I work in, so I can drive by and take care of that stuff), or, worse, from people in some companies that are working remotely with us.

I believe migrations like this should be transparent to the user, and this little detail make them very very opaque.

64 bits hell

Having a full 64 bits OS running on a 64 bits machine can only be a good thing, right? Well… maybe not.

I’m a little crazy and my organizational skills might be very well defined by the word “chaos”, but I’m not crazy enough to do this in the space of two days without having tested all this stuff first and document the important details. So, before trashing our G5 Xserve, I grabbed an old PowerMac G4, installed Leopard Server and all the stuff that really needs to work. The most experienced of you should be smiling by now. Although it seems that the only important difference between both CPUs for the matters we are discussing is just speed, there’s a really important one: 32 bits VS 64 bits. The G5 is a full 64 bits CPU, and the G4 is 32 bits. Up to Tiger, this is not a problem at all, because most of the OS was also running in 32 bits. This included most services, like DBs and Apache. On Leopard, everything (or close to that) is compiled to four different architectures: PowerPC 32 and 64 bits, and Intel 32 and 64 bits. We’ll come back to this in a minute.

Mac OS X Server is bundled with MySQL, PHP and Apache, but not with PostgreSQL. As I prefer PostgreSQL to MySQL by far, I tend to use PostgreSQL with all the applications I can, including my own WebObjects applications. So, I compiled and installed PostgreSQL on the server. As I also need PHP applications to access PostgreSQL databases, I had to download PHP source code and recompile it with PostgreSQL support (you gotta love a language where you have to recompile the whole damn thing to add support to a DB…). But, to compile PHP with support to MySQL (and PostgreSQL) I need to have the MySQL headers and dynamic libraries. Well, Mac OS X Server is bundled with MySQL binaries, but not the headers or libraries. As there were no binaries available for PowerPC 10.5 on the MySQL page, I also had to grab the source and recompile all this stuff.

This is where problems started. I recompiled MySQL, and put it working after some struggle (I really hate MySQL). Then I recompiled PHP. Installed it, added the LoadModule directive to the apache config file, and restarted apache. Bum. Explosions. Apache would not start. It said that the PHP module was compiled for the wrong architecture. I started to thing, WTF, are you telling me that my Xserve just compiled PHP… for Intel? Why did this work on the test G4 box? Well, what other architecture could it be? I started googling for the problem and I got it: apache is compiled for all the four architectures I referred above, and it always runs with the most appropriate one for the machine. In the Xserve case, it uses the PowerPC 64 binaries. The problem is that PHP had been compiled for 32 bits only. Ok, no problem. Go to PHP dir, make clean, poke around with the environment variables, recompile the thing for 64 bits. Bum. More explosions. Guess what, MySQL was NOT compiled for 64 bits! Ok ok, one more level deep in the stack, go to MySQL directory, blablabla, recompile and… BUM! Yet another explosion. Now this one was more complicated. Apparently some of the libraries on the MySQL source code package were not being compiled for 64 bits. So, no 64 bits MySQL means no 64 bits PHP that means no runnable PHP with 64 bits apache that means falling back to Apple’s branded PHP that means… no PostgreSQL.

From what I saw on the Net, convincing MySQL to compile on 64 bits was not a road I wanted to go into. Also, one of the pages I found about the “wrong architecture” problem when starting Apache actually suggested to go in the opposite direction: grab Apache source code and recompile it in 32 bits. Using the mention configure command (./configure –enable-layout=Darwin –enable-mods-shared=all) I compiled the exactly same Apache version that Apple bundles with Leopard Server, and installed it over the Apple branded one. That made it all work, now on 32 bits. Of course, if you follow this trick, please keep in mind that this may break in future system updates. If some Apple system update replaces apache, it will not start unless you recompile it again for 32 bits only, or remove the PHP module.

This 64 bits mess is actually a very nasty problem, and makes me think what I’m actually gaining in all this. And the answer is: zero. My server has one GB of RAM, and will probably never have more than 4. If, for some reason, we actually need to boost the memory so far, it certainly won’t be because of Apache. It gets me thinking about actually how many people will actually need apache to run in 64 bits mode. If it’s more that 1% or 2% of the Xserve users, I’ll be very amazed. And what do I loose? A lot. Not all the open source projects compile easily in 64 bits mode (I know MySQL that comes with Mac OS X Server is compiled for 64 bits, but for some reason the needed fixes for that are not in the public MySQL source code tree), Apache may stop working at all in the next system update, and I had a lot of extra work. Maybe Apple should provide an easy way to switch this kind of stuff between 32 and 64 bits mode at will. Having only one OS version to all the architectures is interesting, but solving the problems that it creates is not.

Wrapping up

Everything is working now, after an entire weekend spent behind many terminal windows. Unfortunately, I have to say that my opinion about Mac OS X Server is not the best one. I have been working lately with FreeBSD. My experience with FreeBSD is way, way less than the experience I have with Mac OS X, so there are probably many downsides in FreeBSD I had not yet to deal with. That being said, I think Mac OS X Server is a very easy to use OS, as long as you keep using the tools Apple provided. As soon as you need different tools, specially the ones that tinker with Apache, you’ll start regretting liking computers in the first place. And surprisingly, you start to find that it’s actually easier to do it in a FreeBSD server. Every software I installed so far in FreeBSD (including WebObjects) was installed in a very easy and straightforward, painless way. Just browse the ports tree, make install clean and there it is. No crazy problems, everything is made to work with everything. And the default configurations are usually safer than Apple’s.

It makes sense: although FreeBSD guys don’t do beautiful GUIs and assistants, they work hard to make sure the system Works. All of it, including all the ports. And most important, not just it works, but it works together. If I had to use a word to define FreeBSD, I would pick “consistency” without hesitation. Even WebObjects, which does not have an “official” port on the FreeBSD port tree actually installs easier in FreeBSD than in OS X (due to the hard work of Quinton Dolan that created a FreeBSD port of WO). And face it: probably all the software you need exist in the port tree. It’s HUGE. And if it doesn’t, you can always install it using the classic UNIX way.

The Apple way is different. Apple picks a very small range of software, compiles and packages it in a very easy to use OS. It’s really easy, way more than FreeBSD in many ways. The problems appear when you conclude that the bundled software is not enough, and you want to install your own. And when that happens, you are completely on your own. You’ll start fighting Apple sometimes weird configurations and file system structure, you may run in binary architecture incompatibilities like I did, and so on. And you’ll probably need to do this, because what comes bundled with OS X Server is probably far from enough to get the job done.

A not very fast but handy way to downscale images in a WebObjects application is using Java 2D APIs, with code like this:


BufferedImage reducedImage = new BufferedImage(newX, newY, BufferedImage.TYPE_INT_RGB);
Graphics2D g = reducedImage.createGraphics();
g.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BICUBIC);
g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
g.setRenderingHint(RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_SPEED);

g.drawImage(originalImage, 0, 0, newX, newY, 0, 0, originalImage.getWidth(), originalImage.getHeight(), null);
g.dispose();


Although not blazing fast, this is enough for many applications. I could reduce a 7 Mega-Pixels image to something like 250 pixels wide in about one second, or less, in my PowerBook G4. But this was in Tiger.

In Leopard, as some of you may have noticed (and if you have applications deployed on Leopard Server, be aware) this is incredibly slow. When I say slow, I say five minutes, or even more, with the CPU being used at 100% during that time.

There are two reasons that lead to this. The first (which is not a problem in itself, but it’s a cause of the problem): Apple switched from Quartz to Sun2D graphics engine as the default one for Java applications on Leopard. So, all your image manipulation is being done using the Sun pipeline now. This would not be a problem, except for the second reason: the Apple JVM implementation has a bug that is slowing Sun’s pipeline drawImage method to a crawl. Actually, that was not the real reason. I testes this on FreeBSD (using Diablo JDK) and the speed was similar to Leopard’s. Sun2D is REALLY slow, to the point of being useless. I’m now using ImageMagick.

The only solution for now is forcing the application to use Quartz engine. You can do that using the command line option -Dapple.awt.graphics.UseQuartz=true. And, of course, file a bug on this!

I really don’t undersand. MS has some talented people working for them. After all they make operating systems, office suites, and many more products. They can’t be that dumb. There must be people there that know what they are doing.

So, why is IE such a big pile of crap? If is it so ridiculously flawed? Why they seem to don’t give a shit? What’s the frickin’ problem about showing a layer with z-index = 5 BEYOND a layer with z-index = 60? WHY DOES THE DAMN BACKGROUND LAYER COVES THE FRONT ONE!? Damn it. Fix IE once for all. Or cancel it’s development. Or adopt Gecko or WebKit or whatever the hell they want. But stop wasting my time. I’m so pissed off with childish and pathetic bugs on IE. Where the hell did the IE team learn to code? Oh wait… did they actually learn to code?!

Today we started hacking some Apache rewrite rules to make some URLs a little more friendly. All of the URLs we are rewriting are entry points for our application, which in the WO world means direct actions.

All of them contain queries parts. These are the arguments in a URL, like http://domain.com/something?arg=value&anotherArg=anotherValue. Well, everything was running fine until we accessed an URL with a escaped character. We had the following rule:


RewriteRule ^/optOut(.*) /cgi-bin/WebObjects/OurApp.woa/wa/optOut$1 [R]


When we tried to access the URL http://domain.com/optOut?email=me%40domain.com, we got:


http://domain.mac/cgi-bin/WebObjects/OurApp.woa/wa/optOut?email=me%2540domain.com


Note the last part or the URL. The problem is that the original % in %40 (the escape code for @) was being itself escaped, leading to a corrupt URL. The result of calling formValueForKey on this was me%40domain.com.

After googling for a while, I found out many people have this problem, but surprisingly I could not find a decent solution. I saw hacks with escape internal functions and PHP weird variables that obviously wouldn’t help me at all, and I was starting to get nervous.

Well, while reading the rewrite module documentation, I found the solution right there: the NE option. According to the docs:

‘noescape|NE’ (no URI escaping of output)
This flag prevents mod_rewrite from applying the usual URI escaping rules to the result of a rewrite. Ordinarily, special characters (such as ‘%’, ‘$’, ‘;’, and so on) will be escaped into their hexcode equivalents (’%25′, ‘%24′, and ‘%3B’, respectively); this flag prevents this from happening. This allows percent symbols to appear in the output(…)

Well, this is it. I saw so many pages with people asking about this problem without getting any answers that I was not very confident on this, but I tried to change the rule for:


RewriteRule ^/optOut(.*) /cgi-bin/WebObjects/OurApp.woa/wa/optOut$1 [R,NE]


And that’s it. It’s now working as it should, producing the correct URL:


http://domain.mac/cgi-bin/WebObjects/OurApp.woa/wa/optOut?email=me%40domain.com


Apparently, it’s that simple.

As written on all the good practice books about development, I write my applications ready for localization. To do that in WebObjects, one should put all the localizable strings in a strings file. In WebObjects applications, a strings file may be written in two formats: the Apple’s weird .plist format and XML. I use XML, because it looks more… well… I don’t really know, but I prefer it.

Anyway, some days ago I did a mistake in one of those files. I did something like: <string>Some String/string>. As you can see, I forgot the < character in the closing tag. When I run the application, all I got was this:


The element type "string" must be terminated by the matching end-tag "“.: Parsing failed in line 1155, column 3


This is totally useless, as the line 1155 is actually the line before the last one in the file (the one that closes the “dict” entity). And yes, it’s a big file. Obviously, checking all the lines manually in undoable, even if I only checked the modified ones after the last commit (I added/changed many strings since then). So, the easiest way was to find a XML validator on the web, and hope it would point me the error.

How surprised I was when all the validators I found in the first two result pages pages of Googling for “XML Validator” reported exactly the same error as WebObjects itself. Great. Just great. And finally, I found one validator that saved the day. This guy, although a bit slow, showed me the exact line where the error was. So, bookmark this. Really. It’s the best XML validator around! And yes, just so that you don’t forget the link, let’s just mention it one more time!

Yesterday I spent all day in the first ever Apple developer event in Portugal. Apple carried out a Leopard Tech Talk in Lisbon, where portuguese developers could learn about some of the new stuff in Leopard, including 64 bit programming and Core Animation. The speakers were splendid, and with great technical knowledge about what they were talking about. It’s always great to watch a technical presentation made by real coders, and not by the full-of-bullshit marketing people.

Some presentations were very superficial, but the most interesting ones went as deep as some of the WWDC sessions I attended. The event was actually a micro-WWDC, and even included a nice buffet with plenty of food for lunch and coffe-breaks, all for free. As I spend almost all the WWDC week on the IT track, it was cool to learn about the desktop stuff Apple is working on.

The room was packed, and people were motivated and participative. The Apple guys really liked that, as they say those are the main factors they use to evaluate how successful and event it, specially when going to a new country for the first time. I know I’ll be there next time!

Recently I had to profile a WO application for memory leaks (more exactly, unintentional retention of objects that prevented them from being garbage-collected). Some quick notes:

• Eclipse TPTP will not work on Mac OS X. Period. Although Eclipse allows you to install it without complaining, it does not work at all.
• I found this nice article about profiling and trouble-shooting in Java. Page 37 describes how to do what I initially wanted, although I didn’t need to get that far.
• There are a lot of nice tools to help you peek inside the environment of your Java applications right there on your hard drive. I used jconsole. It’s really easy to use. First, add the -Dcom.sun.management.jmxremote VM option when launching the application. On Eclipse, this would be the “VM Arguments” in the “Arguments” tab of the Run Panel. Second, just open a terminal and write “jconsole”. Attach to the running app, and have fun.
• Instruments (aka xRay) on Leopard has only one Java instrument, Java Threads. It appears to be almost useless on Java profiling. Yet another reason to stay on Tiger for now.