drawImage performance on Leopard

February 27th, 2008

A not very fast but handy way to downscale images in a WebObjects application is using Java 2D APIs, with code like this:


BufferedImage reducedImage = new BufferedImage(newX, newY, BufferedImage.TYPE_INT_RGB);
Graphics2D g = reducedImage.createGraphics();
g.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BICUBIC);
g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
g.setRenderingHint(RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_SPEED);

g.drawImage(originalImage, 0, 0, newX, newY, 0, 0, originalImage.getWidth(), originalImage.getHeight(), null);
g.dispose();

Although not blazing fast, this is enough for many applications. I could reduce a 7 Mega-Pixels image to something like 250 pixels wide in about one second, or less, in my PowerBook G4. But this was in Tiger.

In Leopard, as some of you may have noticed (and if you have applications deployed on Leopard Server, be aware) this is incredibly slow. When I say slow, I say five minutes, or even more, with the CPU being used at 100% during that time.

There are two reasons that lead to this. The first (which is not a problem in itself, but it’s a cause of the problem): Apple switched from Quartz to Sun2D graphics engine as the default one for Java applications on Leopard. So, all your image manipulation is being done using the Sun pipeline now. This would not be a problem, except for the second reason: the Apple JVM implementation has a bug that is slowing Sun’s pipeline drawImage method to a crawl. Actually, that was not the real reason. I testes this on FreeBSD (using Diablo JDK) and the speed was similar to Leopard’s. Sun2D is REALLY slow, to the point of being useless. I’m now using ImageMagick.

The only solution for now is forcing the application to use Quartz engine. You can do that using the command line option -Dapple.awt.graphics.UseQuartz=true. And, of course, file a bug on this!

I hate IE

February 1st, 2008

I really don’t undersand. MS has some talented people working for them. After all they make operating systems, office suites, and many more products. They can’t be that dumb. There must be people there that know what they are doing.

So, why is IE such a big pile of crap? If is it so ridiculously flawed? Why they seem to don’t give a shit? What’s the frickin’ problem about showing a layer with z-index = 5 BEYOND a layer with z-index = 60? WHY DOES THE DAMN BACKGROUND LAYER COVES THE FRONT ONE!? Damn it. Fix IE once for all. Or cancel it’s development. Or adopt Gecko or WebKit or whatever the hell they want. But stop wasting my time. I’m so pissed off with childish and pathetic bugs on IE. Where the hell did the IE team learn to code? Oh wait… did they actually learn to code?!

Avoid escaping URLs in Apache rewrite rules

January 24th, 2008

Today we started hacking some Apache rewrite rules to make some URLs a little more friendly. All of the URLs we are rewriting are entry points for our application, which in the WO world means direct actions.

All of them contain queries parts. These are the arguments in a URL, like http://domain.com/something?arg=value&anotherArg=anotherValue. Well, everything was running fine until we accessed an URL with a escaped character. We had the following rule:


RewriteRule ^/optOut(.*) /cgi-bin/WebObjects/OurApp.woa/wa/optOut$1 [R]

When we tried to access the URL http://domain.com/optOut?email=me%40domain.com, we got:


http://domain.mac/cgi-bin/WebObjects/OurApp.woa/wa/optOut?email=me%2540domain.com

Note the last part or the URL. The problem is that the original % in %40 (the escape code for @) was being itself escaped, leading to a corrupt URL. The result of calling formValueForKey on this was me%40domain.com.

After googling for a while, I found out many people have this problem, but surprisingly I could not find a decent solution. I saw hacks with escape internal functions and PHP weird variables that obviously wouldn’t help me at all, and I was starting to get nervous.

Well, while reading the rewrite module documentation, I found the solution right there: the NE option. According to the docs:

‘noescape|NE’ (no URI escaping of output)
This flag prevents mod_rewrite from applying the usual URI escaping rules to the result of a rewrite. Ordinarily, special characters (such as ‘%’, ‘$’, ‘;’, and so on) will be escaped into their hexcode equivalents (’%25′, ‘%24′, and ‘%3B’, respectively); this flag prevents this from happening. This allows percent symbols to appear in the output(…)

Well, this is it. I saw so many pages with people asking about this problem without getting any answers that I was not very confident on this, but I tried to change the rule for:


RewriteRule ^/optOut(.*) /cgi-bin/WebObjects/OurApp.woa/wa/optOut$1 [R,NE]

And that’s it. It’s now working as it should, producing the correct URL:


http://domain.mac/cgi-bin/WebObjects/OurApp.woa/wa/optOut?email=me%40domain.com

Apparently, it’s that simple.

A decent XML validator

January 20th, 2008

As written on all the good practice books about development, I write my applications ready for localization. To do that in WebObjects, one should put all the localizable strings in a strings file. In WebObjects applications, a strings file may be written in two formats: the Apple’s weird .plist format and XML. I use XML, because it looks more… well… I don’t really know, but I prefer it.

Anyway, some days ago I did a mistake in one of those files. I did something like: <string>Some String/string>. As you can see, I forgot the < character in the closing tag. When I run the application, all I got was this:


The element type "string" must be terminated by the matching end-tag "“.: Parsing failed in line 1155, column 3

This is totally useless, as the line 1155 is actually the line before the last one in the file (the one that closes the “dict” entity). And yes, it’s a big file. Obviously, checking all the lines manually in undoable, even if I only checked the modified ones after the last commit (I added/changed many strings since then). So, the easiest way was to find a XML validator on the web, and hope it would point me the error.

How surprised I was when all the validators I found in the first two result pages pages of Googling for “XML Validator” reported exactly the same error as WebObjects itself. Great. Just great. And finally, I found one validator that saved the day. This guy, although a bit slow, showed me the exact line where the error was. So, bookmark this. Really. It’s the best XML validator around! And yes, just so that you don’t forget the link, let’s just mention it one more time!

Leopard tech talk, Lisbon

December 4th, 2007

Yesterday I spent all day in the first ever Apple developer event in Portugal. Apple carried out a Leopard Tech Talk in Lisbon, where portuguese developers could learn about some of the new stuff in Leopard, including 64 bit programming and Core Animation. The speakers were splendid, and with great technical knowledge about what they were talking about. It’s always great to watch a technical presentation made by real coders, and not by the full-of-bullshit marketing people.

Some presentations were very superficial, but the most interesting ones went as deep as some of the WWDC sessions I attended. The event was actually a micro-WWDC, and even included a nice buffet with plenty of food for lunch and coffe-breaks, all for free. As I spend almost all the WWDC week on the IT track, it was cool to learn about the desktop stuff Apple is working on.

The room was packed, and people were motivated and participative. The Apple guys really liked that, as they say those are the main factors they use to evaluate how successful and event it, specially when going to a new country for the first time. I know I’ll be there next time!

Profiling WebObjects applications

November 30th, 2007

Recently I had to profile a WO application for memory leaks (more exactly, unintentional retention of objects that prevented them from being garbage-collected). Some quick notes:

  • Eclipse TPTP will not work on Mac OS X. Period. Although Eclipse allows you to install it without complaining, it does not work at all.
  • I found this nice article about profiling and trouble-shooting in Java. Page 37 describes how to do what I initially wanted, although I didn’t need to get that far.
  • There are a lot of nice tools to help you peek inside the environment of your Java applications right there on your hard drive. I used jconsole. It’s really easy to use. First, add the -Dcom.sun.management.jmxremote VM option when launching the application. On Eclipse, this would be the “VM Arguments” in the “Arguments” tab of the Run Panel. Second, just open a terminal and write “jconsole”. Attach to the running app, and have fun.
  • Instruments (aka xRay) on Leopard has only one Java instrument, Java Threads. It appears to be almost useless on Java profiling. Yet another reason to stay on Tiger for now.

Submitting a form with target set to a script-generated iframe on IE

November 2nd, 2007

(nice title, isn’t it?)

Although AJAX techniques are now widespread among the web-developers community, there’s still something that can’t be done using pure AJAX techiques (whatever that means): file uploads. There’s no way to grab a file using JavaScript and send it to the server using an asynchronous request. I guess the main reason for this is security, you don’t want web sites to steal files from your home directory.

Anyway, people found a relatively popular way to do this. The ideia is having a hidden iframe on the page, and use that iframe as a target of the form where you have the reference to the file to upload. So, when the form submits, the result page will go to the hidden iframe, and the user won’t see a full page refresh. You still have to take care of some details, like hiding the form and showing a nice progress bar with the classic “Hold on, we are uploading” message, and polling for the content of the iframe to check is the file is still in its way, or if it arrived safely.

One nice detail is where do you actually have the hidden iframe. You could just put the iframe on the HTML code of your page, but IMO that’s ugly. The iframe is an artifact that is needed just to serve as a black hole for the form submittion result page. It doesn’t make sense to put it in your HTML code, as it hasn’t anything to do with the content. Also, it’s error prone: you may inadvertently delete it, causing the file upload to missbehave. Or you may need to change the iframe code and having it on HTML will force you to update on all the pages where you use it (and of course, you will forget one).

So, I believe the most elegant solution (if you can use the word “elegance” in the context of this major hack) is to generate the iframe using javascript. You may do this on the page load, when you create the form, whatever. Just do it before the user submits the file! :) Well, it’s easy, right? Something like this does the trick:


var objBody = document.getElementsByTagName("body").item(0);

var iframe = document.createElement('iframe');

iframe.id = 'fileUploaderEmptyHole';
iframe.name = 'fileUploaderEmptyHole';
iframe.width = 0;
iframe.height = 0;
iframe.marginHeight = 0;
iframe.marginWidth = 0;

objBody.insertBefore(iframe, objBody.firstChild);

That’s cool, right? Just run this and the iframe will be created. Submit the form, the result goes in the hidden iframe, everything works, we are done, let’s go home. Well… all true until you actually test it on Internet Explorer…

If you test this on IE, the result will be a new window being created when you submit, which is clearly not what you want. Well, I had a hard time finding the problem, so here goes: if you create the iframe using JavaScript, IE wont set it’s name. Yes, the “iframe.name = ‘fileUploaderEmptyHole’;” line will simply be ignored. It does nothing. So, as you don’t have any frame called ‘fileUploaderEmptyHole’, when you submit the form, it will create a new window named ‘fileUploaderEmptyHole’ and display the result on it.

The solution it to hack this even further. Specifically:

iframe = document.createElement('<iframe name="fileUploaderEmptyHole">');

Yeah! Now you’re thinking “WTF?”. Yes, yes, it’s true. This actually works on IE, with the expected (?) results. Well, you still have to support the other browsers, but you are lucky, as this will throw an exception on all the non-IE browsers. So, it’s just a matter of catching it, and running the decent version of the code:


var iframe;
try {
  iframe = document.createElement('<iframe name="fileUploaderEmptyHole">');
} catch (ex) {
  iframe = document.createElement('iframe');
}

iframe.id = 'fileUploaderEmptyHole';
iframe.name = 'fileUploaderEmptyHole';
iframe.width = 0;
iframe.height = 0;
iframe.marginHeight = 0;
iframe.marginWidth = 0;

This is why I love the Web. Or maybe not.

Welcome to the Leopard WebObjects mess

October 30th, 2007

So, you thought Leopard and WebObjects 5.4 would bring some good news to WebObjects developers, right? Well… so did I.

I’m not even going to talk much about Leopard itself. It’s clearly an unfinished product, with dubious interface “improvements”, and totally unstable, at least on PowerPC macs. I try to keep my blog tecnhically-centered, and not based on personal opinions, but please Apple… you can do much better than that. Stop playing arround with the iPhone and concentrate on what really matters.

Well, about WO… Leopard has a new version of WebObjects, 5.4. It has some bug fixes and improvements over the previous versions, but unfortunately there are still a lot of show-stoppers that keep me from installing Leopard (and as you can guess from the last paragraph, I’m not specially motivated for that).

The first one is that Wonder still does not compile on 5.4. So, if you use Wonder (and you should), you must bring WebObjects 5.3 to Leopard. It’s not a very complicated process, but of course, it means trashing the brand new version. The good news is, if you use Wonder, you don’t really need WO 5.4, because almost all the features are already implemented on Wonder. The ideia is installing 5.4 (basically to get the Apache 2.2 adaptor), them trashing everything else, and finally installing 5.3. I installed it from the xCode 2 DMG and it worked fine. Anyway, Milke did an excellent (as always) document about his trip back to 5.3 on Leopard.

You should note the detail that the plists for LaunchD to boot WOTaskD and WOMonitor are not even in Leopard Client. Apparently, someone actually forgot to include that in the release. Assuming that most people develop without direct connect enabled (I don’t know how true this assumption is, but at least I do!), implying that they actually need WOTaskD to be running, it’s a bit scarry to see that actually no-one at apple tested this, or else they would find out that the files were not included. It surely doesn’t boost my confidance on all this.

Also, I develop using virtual hosts. Basically I set the adaptor URL, in WOLips launch options, to be something like http://aVirtualHost.mac/cgi-bin/WebObjects. I kept getting the “No instance available” message until I run “sudo hostname -s aVirtualHost.mac” on the command line and restarted every WO-related process (apache, WOTaskD, and the app itself). BTW, note that the file included in apache2 configuration for vhost configuration has errors, and will actually kill apache at launch if you don’t remove both examples that come with it (the error is in the logging configuration for both sample virtual hosts). Even worse, if you are trying to look for apache error logs in the error_log apache file as you always did on tiger… forget it. This kind of errors are now thrown to /var/log/system.log.

But the biggest show-stopper is EOGenerator. You do use EOgenerator, don’t you? EOGenerator was based on the same frameworks as the Apple WO tools, like EOModeler. Apple moved those frameworks from “deprecated” to “dead” with Leopard, so EOGenerator went to the grave with them. Relax, there will be a Java replacement included on WOLips, but it’s still not there. Well, talk about living in the information age… the EOGenerator is out! :)

So, if you are an WebObjects developer wondering about jumping to Leopard… my adviced is, wait. Unless you don’t use Wonder and EOGenerator, which is highly unlikely.

onBeforeUnload Javascript BODY attribute

September 14th, 2007

Those who code Javascript know that the BODY tag has two cool attributes where we can hook methods to be run when the page loads and unloads. Those are the standard onLoad and onUnload attributes.

onLoad does what you expect - runs whatever the attribute is set to after loading the entire page (or, at least, all the HTML and Javascript files).

onUnload is a little more tricky. When you have a function hooked to onUnload attribute, and you click on a link, the function will actually be executed after the browser opens the request to get the new page. This can be catastrophic in many ways. Imagine that you are on a page where you can edit some information in an asynchronous way. You edit all the stuff you want, and the browser only sends the data to the server from time to time, and when you leave the page.

Now imagine that the next page reflects the data changes you just did. You would have some kind of AJAX call on the onUnload attribute that would send the final changes to the server. But these changes will be sent after the browser opens the http request to get the next page, and actually get it from the server. You got a problem: the next page won’t reflect those last updates because it was generated before they were sent to the server!

Happily, there is a solution. Someone (I think Microsoft, but I’m not sure) implemented the onBeforeUnload event. As the name suggests, this is a hook to a method that will be executed before the page unloads, and before the http request to get the next page is open. This is a non-standard attribute, but it’s implemented on all the major browsers (IE, Firefox and Safari). And it’s really useful!

There’s a small problem with IE and tabs that you have to handle. I didn’t test this on IE yet, but this guys tell you all about it.

Recovering from optimistic locking exceptions

August 17th, 2007

WARNING: the solution presented in this article is wrong. It’s a partial solution only, it won’t always work, and you may screw up your data. To see why, check my second article on this issue.

There are many situations, in simple web applications, where you don’t really have to deal with concurrent data update. Imagine you are writing a blog app, like WordPress. Even if two persons are changing a post title at the same time (which, by it’s nature, it’s a really rare event), you, as the developer, may simply don’t care about it, and rely on the “last write wins” strategy.

When you are writing more complex stuff, specially when one needs to have really complex data being modified by different users, in different situations, at the same time, and when you have pre-calculated data, because grabbing all the objects and calculating the results in real time (fast enough for a page load) is simply not possible, you will have serious headaches (it’s already hard enough to understand this sentence without re-reading it!).

When developing WebObjects, there are two main classes of concurrency problems to solve. Intra-instance concurrent data updates, and inter-instance concurrent updates. It would appear that the second is much harder than the first… well… it’s not.

Intra-instance updates should be simplified by automatic data merging between contexts. But there are some problems. The biggest problem is that, in fact, it’s possible to totally overwrite a data modification done by another user without getting any notification of that at all. How? See this thread on the WebObjects Dev mailing list. The biggest problem is that, while a context is locked, data merging simply does not occur. So, imagine you lock context A and start modifying data. At the same time, someone locks context B on another thread, changes the same data that you are modifying in context A, and saves it. The data you changed is merged on all the other contexts where the same objects are present, as long as they are unlocked. But remember, your context A isn’t, so no merging will occur for now. Now, imagine you continue working on context A, and finally save it, and unlock it. The merging will occur now - to late, because you already saved, and probably won’t work any more in that context.

By now, you say “Naaah, that won’t happen, because WebObjects will trigger an optimistic locking exception”. No it won’t. Why? Remember that context B saved while context A was locked, right? So, when B saves, the row snapshots of the EOF stack will be updated. When you save context A, assuming that no one else (besided the A and B guys) worked on the same data at the same time, EOF will base the OL check on the row snapshots that is has in the stack. Remember, this are no more the snapshots for objects in context A. They are the snapshots caused by B saving, and those will be the same that exist in the DB. So, no OL exception will occur. Congratulations, A just trashed B data updates, and wasn’t notified at all.

Are you already scared? Good. So be even more scared. There are only two solutions for this. The first one, create a lot of separate EOF stacks (one per session, assuming the sessions are being locked in the normal WO way). This sucks, because it’s too resource expensive. It will use a lot of memory (lots or repeated snapshots in memory) and it will open many connections to the DB, that may be a problem by itself. The second solution… classic Java locking (syncronized keyword, etc). This is the time you start thinking about being a farmer, right?

Well, the news on inter-instance concurrency are better, although far from perfect. Here, there’s no data merging, and no Java syncronized stuff. Everything will be based in locking. Usually optimistic locking, but you can also use pessimistic locking (ie, “real” row-database locking). Pessimistic locking has huge problems and many experienced coders will recommend that you stay away from it. So I will not cover that, and I’ll assume you want to use Optimistic locking.

The ideia is simple: create context, fetch data, modify data, save. If you get a locking exception, re-fetch data, re-modify, and re-try to save it until no OL exception is thrown.

This is simple for simple stuff, ie, when you have one single object causing the failure. Now imagine the following scenario (and this isn’t imaginary, it occured to me): you are working with dozens of objects thay may cause OL exceptions, and if one of them causes it, all the others will probably cause too. Also, I need to create some objects (relationships, mostly, but also some individual objects) depending on what I’m doing. If I get an OL exception, I must go back, delete those objects, and create new ones, because the fact that I create, or not, an specific object depends on the data already present on the data storage. In theory, this will not be a problem: create context, fetch data, modify data, create objects (and save the created objects in a temporary array), save, get OL exception, delete all the created objects, re-fetch data, re-modify data, re-create objects, save. Right? Wrong. Unfortunately I’ve hit some obscure WebObejcts bug, as you can see in this WODev thread. This is the time you start browsing the net looking for a farm to buy, if you don’t already have one.

After a lot of experiencies, I think I have reached a reasonable way to handle this in a simple manner (remember, you will always have to think about intra-instance problems, but we are dealing with inter-instance now). Instead of fighting with WO about deleting created objects, and getting outdated data when one was supposed to get fresh one, simply don’t worry about it, and create everything from scratch every time you try to save. The code will look something like this:

synchronized (lock) { // Solving intra-instance problems,
                      //your millage may vary on this, of course
  int tries = 0;

  // Make MAX_SAVE_ANSWERS_TRY_COUNT something reasonable, like 50
  while( tries < MAX_SAVE_ANSWERS_TRY_COUNT ) { 

    // Setup context
    EOEditingContext context = new EOEditingContext();
    context.setFetchTimestamp(System.currentTimeMillis());

    // Make sure nothing "strange" will happen, this is a simply delegate
    // that blocks merging. Probably it won't be needed but I'm paranoid.
    context.setDelegate(new NoMergingECDeletage());

    // Register the EC in my session lock manager, to handle locking and
    // garbage collection automatically. Your millage may vary, specially
    // if using Wonder in a smarter way than I do.
    session.lockManager().registerEditingContext(context); 

    // Get local copies of objects
    < here you get all the local copies of your objects. Remember the
    context.setFetchTimestamp(System.currentTimeMillis()); line above?
    This will guarantee that ALL the objects you get in this context will
    contain fresh data. You can use EOUtilities.localInstanceOfObject,
    walking trough relationships like object.otherObject().anotherOne(),
    using fetch specifications, whatever. Everything will be fresh. >

    // Let’s do it.
    < Do your thing here. Create objects, modify objects, delete objects,
    go crazy. >

    // Try to save
    try {
      context.saveChanges();
      return;
    } catch (EOGeneralAdaptorException saveException) {
      ++tries;

      // isOptimisticLockingFailure is basically the method
      // with the same name in Apple docs
      if( Util.isOptimisticLockingFailure(saveException) ) {

        NSLog.out.appendln(”Optimistic locking exception”);
        // Note that I don’t refault anything here. I don’t need it.
        // On the next iteration, a new EC will be created, and all
        // the objects will be automatically refaulted.
      } else {
        // It’s some other exception, handle it somewhere else
        throw saveException;
      }
    }
  }

  throw new RunTimeException(”Could not save after ” +
MAX_SAVE_ANSWERS_TRY_COUNT + ” tries. Help me.”);
}

At the beggining, I didn’t like this approach at all. But after using in some places, and seeing it work perfectly, I’m liking it more and more. The main advantage is, it works. No strange problems, no fighting with WO bugs, no strange refaulting behaviour. It’s a dream come true. It just works. But, of course, it has some disadvantages. All the objects are fetched from the data store, with the consequent performance hit. So, if you are dealing with many many objects, this may be undoable for you. Also, it has a more serious problem if you are lazy like me: you cannot have an object binded to the page component, and use it in the processing. The reason is that you need to bring that object “inside” the newly created EC, and localInstanceOfObject won’t copy the unsaved modifications, and will not even copy newly created objects. So you have to manually copy and reproduce all the user modifications to the local copies of objects. More work for you, more work for the CPU, more objects in memory. It’s life.

Written at the beautiful and peaceful town of Serpa. Not on an iPhone.