Terminal.app

Pascal Robert has just setup an online survey for WebObjects developers. The survey is short and direct2thePoint so you won’t spend more than 5 minutes on it. The results will be delivered to Apple, to help them focus their development resources on the needs of the community. As Pascal said, the rules are:

• - One survey per organization, if I see the same organization more than one time, I will delete the duplicates
• - Please, please, be honest in your answers

So, if you are an WebObjects developer, take the survey now!

As you may know, Safari 3.0 finally fully supports SVGs. Based on that, Ravi Mendis released the first beta of svgobjects 2.0, an SVG framework for WebObjects. Apparently, the framework started with the goal of drawing charts dynamically, but the author extended it to support the drawing of full-screen interfaces that could be used on the desktop and, even more, on mobile devices, like the iPhone. It may become a very useful framework for building iPhone-targeted web apps!

Well, WWDC is almost done, I’m waiting for the last session on Level 3 of Moscone, near one of these really big windows.

It’s a really big pain to not being able to write here all that I have learned about WebObjects (for those of you who don’t know, WWDC sessions are covered by NDAs).

Someone asked on the WebObjects mailing list how do we classify the WO news, from A to C. Well, I can say, A++++! I think the future of WebObjects will be susprising. Apple showed us some stuff that, in my opinion, is probably almost as revolutionary as the original WebObjects framework itself. But we still will have to wait.

The only thing I can say more is: if you are (considering) doing web application development, don’t be stupid - learn WebObjects NOW. Currently, it’s the most powerful web application development environment by far. And in some future time this advantage will be an order of magnitude higher.

I’ll go tomorrow to SF, to assist WOWODC 2007 and WWDC 2007, so I won’t post here for some time.

WOWODC was organized by Pascal Robert, as a result of the lack of support Apple gives to WebObjects developers, and will feature some well-known people in the WebObjects community, like Chuck Hill, Mike Schrag or Anjo Krank, who we will certainly flood with questions. Mr. Pierre Frisch, the WebObjects Product Manager at Apple will be there too. This will be a great year for WebObjects, as it’s community will provide the attention it deserves!

See you in two weeks.

When developing a Web Application, many times you need to test how will it react then operating over a slow internet connection, like 56k modem, GPRS or any other situation where the bandwidth is tight. This is important not only to test how long will those funky pictures take to load, but as AJAX gets more and more used everywhere, you really need to guarantee that everything works acceptably on slow network links, including AJAX calls, timeouts, animations, and everything else that might depend on the server.

There are some graphical applications for Mac OS X that try to do that, but many don’t work as promised. Well, Mac OS X has the solution built-in, you just have to configure it, and that it the internal firewall (ipfw). Since the introduction of Tiger, ipfw can now do QoS (Quality of Service) stuff, and that means you can create channels (or pipes, in ipfw terminology) and control the priority, bandwith, delay and some other settings of those channels. Than, you simply have to pump the traffic you want to see slowed down thru those channels.

I have done two simple scripts to configure my firewall automatically. They will slow down the http port (80) to a crawl. I do not handle the SSL port for https traffic, but it’s easily achieved by duplicating the script content and adjusting the parameters.

Here are the scripts. The first one, that slows down everything:


#!/bin/sh

/sbin/ipfw add 100 pipe 1 ip from any 80 to any out
/sbin/ipfw add 200 pipe 2 ip from any 80 to any in
/sbin/ipfw pipe 1 config bw 128Kbit/s queue 64Kbytes delay 250ms
/sbin/ipfw pipe 2 config bw 128Kbit/s queue 64Kbytes delay 250ms


And the one that brings your network back to normally:


#!/bin/sh

/sbin/ipfw delete 100
/sbin/ipfw delete 200


As you can see, this slows down all your http traffic to 128 Kbps, and introduces a delay of 250ms (which is roughly what you have on slow, modem-like or wireless connections). The delay can really impact the performance of your application, specially if you do a lot of requests to get information to display on a web page. You can change the values accoring to your needs, and even bring the 128 Kbps down to 56 or even less if you have the guts! I still don’t know exactly what infuence is caused bu changing the queue size, but the 64 KB value produces a nice result.

Don’t forget, if you are just making a static web page, for this to work, you have to turn on the apache web server (the easiest way if to turn on Web Sharing in the System Preferences “Sharing” panel) and browse your pages thru it. Direct file access won’t naturally be affected by the firewall. Also, this slows down ALL your http traffic. Not only the traffic that comes from your computer, but also the traffic that comes from the Internet. So, if you are wondering why is your net really slow, the solution is simple: you forgot to run the second script after finishing your tests! If for some reason you cannot recover the normal speed, just reboot and your Mac will be back to normal.

To run this scripts, just copy them to two text files (using any text editor that saves in TXT - RTF won’t do it!) and then make then executable (using the “chmod u-x ” command on the shell). Also, you have to run then as root (using “sudo” or switching to root before running them).

I use Retrospect to backup most of the machines at GAEL. You may wonder why do I use a commercial tool that still shows it’s OS 9 roots, instead of open source alternatives. Well, Retrospect has some cool advantages (namely the very good support of laptops that may be disconnected abruptely from the network while a backup is in progress). Also, when I first did this setup, Amanda and other tools did not work reliably with Mac OS X file format.

While this works to backup all the desktop workstations and laptops of GAEL members, I have a problem with our xServe. It runs Mac OS X Server, and Retrospect will not backup machines with the Server version of Mac OS X with the license we have. To do that, we would have to buy a much more expensive license.

No problem. A server, due to it’s nature, doesn’t have the “sudden disappearing” problem of the laptops, so I can use a “classic” UNIX approach - and my choice was rsync and the –link-dest option. You may read about this option in the rsync manpage, but in case you don’t know, what it does is the following: instead of synchronizing a directory in the usual way, it will create a new directory with a new file tree. But, to save space, it won’t copy the non-updated files from the old tree to the new one. Instead, it creates hard links, so that both entries in the file system point to the same data on the hard drive (to the same inode), thus saving space. So, everytime you update your backup, you will create a new tree, but you will only waste the space required by the files that were updated since the last backup, and some more space for the filesystem structures that support the directory tree. You can use a command like this:


// rotate old dirs
rm -rf /Volumes/Storage/test/test.5
mv /Volumes/Storage/test/test.4 /Volumes/Storage/test/test.5
mv /Volumes/Storage/test/test.3 /Volumes/Storage/test/test.4
mv /Volumes/Storage/test/test.2 /Volumes/Storage/test/test.3
mv /Volumes/Storage/test/test.1 /Volumes/Storage/test/test.2
mv /Volumes/Storage/test/test.0 /Volumes/Storage/test/test.1

/usr/bin/rsync --rsync-path=/usr/bin/rsync -az -E -e ssh --exclude=/dev/\* --exclude=/private/tmp/\* --exclude=/Network/\* --exclude=/Volumes/\* --exclude=/private/var/run/\* --exclude=/afs/\* --exclude=/automount/\* --exclude=/.Spotlight-V100/\* --link-dest="/Volumes/Storage/test/test.1" "root@my.machine.com:/Users/arroz/TestDirectory" "/Volumes/Storage/test/test.0/"


Side note: the -E option (capital E) is an option present on Mac OS X rsync version, that forces rsync to copy all the extended Mac file system attributes, including resource forks. It only exists in Mac OS X 10.4 (Tiger) or newer versions. If you are still using 10.3 (Panther) or older, use rsyncx. Do not use rsyncx with Tiger.

Until about a week ago, my backup machine (an old PowerMac G4) had an external SCSI Raid with 640 GB, and an internal RAID 0 (2 * 80 GB drives), besides the boot disk. All the Retrospect backups were being placed on the external RAID, and the server backups were going to the internal RAID 0. Now, I know it’s living on the edge to backup to a RAID 0. But there was really no more space, and it was a temporary situation, because the new drives for the external RAID were already ordered.

When the new drives arrived, I stored all the backups where I could for some days (640 GB was huge when we purchased the RAID, but today is relatively managable), switched the drives and created a new fresh RAID 5. Formatted it in the HFS+ file system, and copied back all the backups, including the server backups and finally trashed the internal RAID 0.

Some path adjustements on my server backup scripts, and we are back in business. But the RAID free space was getting dramatically shorter every day. I used the ‘ls -i’ command to compare the inodes of files that were supposed to be unchanged from the backup of a day to the other in the next day, and as I suspected, rsync was duplicating all the files, instead of hard-linking them.

After Googling a lot, I could not find answers for this. I tried to see if ‘cp -la’ would successfully create hard links, but to my surprise, I found out that the Mac OS X built-in ‘cp’ command would not support the “l” option. Nice. Before installing the GNU ‘cp’ version (and because I’m lazy and I didn’t want to do that) I started thinking about everything I had done since the new drives arrived. The OS was the same, the rsync command was the same, it worked before, so it had to work now. The only reason why it could be not working was because rsync, somehow, thought that all the files were changing, even when they did not.

Suddenly, the solution poped up in my head. Mac OS X has an option, associated to every HFS+ volume, called “Ignore ownership on this volume”. This is turned off by default on the boot drive, but it’s turned on by default on all the external drives you format. There’s a good reason for this: Mac OS X is a consumer product. And average users want to buy an external drive, store data on it, bring it to another Mac, and read their data. They don’t care if their UID is the same on both machines or not.

But this causes serious problems to rsync. Althought the file system will store the owner of the files, it probably won’t report it to the applications who try to read it (or will mask them to the user who’s trying to access them). Somewhere this information is filtered, between the file system and application layers. So, rsync was not getting the real UID of the files. As the files that came from the server had real UIDs, both UIDs wouldn’t match, and rsync would create a new copy because, from it’s point of view, the file had been changed.

The solution was simple - just going to the machine console, and “Get Info” of the external volume. I turned off the “Ignore ownership on this volume” setting, and rsync started operating normally again.

Welcome to Terminal.app!

This blog will be dedicated to everything “non-I” in the Mac world. People usually relate Apple to successful consumer products, like the iPod, the iLife suite, and so on. But there are many Apple products that do not pop-up in the stores shelves. Stuff like Mac OS X Server and WebObjects, made for serious business, enterprise applications, network management, and so on.

I work as a system administrator at GAEL, and I’m an independent software developer. I mainly use WebObjects to build my web applications. It’s a relatively unknown, but very powerful and mature framework, born in the NeXT labs, and actively maintained by Apple.

Here, I’ll write about some non-trivial stuff that I learn during my work. Hacks, solutions to (un)common problems, good practices, and so on. I always liked to share my knowledge, and this is the way I’m going to do it from now on. I hope you find it useful!