Terminal.app

PGP is about to release a new version of it’s PGP Whole Disk Encryption product. This version will fully support the Mac, including the boot disk. I’m looking for a full disk encryption system for a long time now, as it’s a really nice solution in a world where the number of stolen laptops keeps increasing. This won’t help at all recovering a stolen Mac, but at least gives you peace of mind about the data in your stolen computer drive. The drive contents will appear as random data for anyone who doesn’t own the PGP password, so you know that, despite having to buy another laptop, your data won’t be seen by anyone else. As we carry more and more vital data inside our laptops (not just the data itself, but also passwords and ssh keys that allow access to servers with even more important data), protecting all that stuff in case of theft of loss is becoming more important every day.

I hope they release a demo version, as I would like to know how the system reacts when awaking the machine from sleep and using target mode. Until then, you can register yourself at the PGP site to be notified on the release date.

Well, I’m back from another WOWODC and WWDC. I’m still really tired, but some quick notes:

• As David LeBer already mentioned, Pascal did an amazing job organizing WOWODC all by himself. Great room (a bit cold on the first day ), large windows and sunlight on the halls. Food (well… not that good, but after all, it’s USA!) and caffeine provided frequently. Very very nice. Suggestion for the next year: a bigger (and brighter) screen, and eventually plasma screens among the room to make it easy for people in the back to read the code.
• I learned a lot about WO frameworks out there (like Wonder, Houdah and specially LEWOStuff that I did not know before). I met for the first time some very talented people, and of course, all the folks from the previous conferences. It’s great to be able to have technical discussions and know different views on the same problems from all those skilled and experienced people out there, face to face.
• WWDC had some interesting news on many stuff. As you know, I cannot talk about the stuff under NDA, so I shall only say that some interesting stuff is being done on the WO side. Also, as you all know by now, the iPhone is now 3G, includes a GPS, the price was slashed, and will be available in many countries of the world. I just hope the service providers slash the data roaming prices, because that makes the iPhone useless when you go to foreign countries. Finally, Snow Leopard was announced, and, as already expected, the focus is not on new features, but on a big cleanup of the OS infrastructure. Not only this are great news for us, developers, but also shows some courage from Apple and a lot of respect for their users. They want to focus the next year on improving the quality of their OS, rather than packing it up with some new features just to win the race against the competition.
• As a side note, the MacTech people was giving away some magazines for free to the people who were standing in line during the morning. I took the time to read most of it during my flight, and I really liked it. I was a MacTech subscriber in the past, but I cancelled it because, during my graduation, I didn’t have time to read it (it’s good to graduate on a place where you actually don’t have time to learn, isn’t it?). Maybe I’ll subscribe it again now.
• Not related to the conferences themselves, we went to visit the bay area surroundings on Friday afternoon. We did the classic trip to the Apple and Google campuses, because we are all geeks, but we also went to the Stanford and Berkeley campuses. The Standford campus totally blow me away. You have to see it to believe it. From now on, I’ll laugh, really laugh, every time I hear a faculty from my university stating that we actually have a campus. The Berkeley campus did not impress me much. It’s more urban style, more crowded and dense. I prefer the Stanford way, with space, a huge amount of space, tons and tons of space, really. Almost made me want to return to the univ!

I finally moved to an Intel machine. Despite the dramatic speed improvement in everything Java-related, namely Eclipse, there’s another big advantage: being able to run IE on Windows using a virtual machine. Unfortunately, that’s something every web developer must do to ensure his or her application will work on the most used (and crappy) browser on earth.

I installed Parallels and created two virtual machines, one for IE 6 and another one for IE 7. This way I’m sure there are no weird problems between those two versions (having more than an IE version on Windows can only be accomplished by hacks, and hacks are bad). Also I can install Visual Web Developer Express Edition on each of the VMs, and use either IE 6 or 7 to debug.

My apps run inside virtual hosts on Mac OS X apache, under a fake DNS name. On Mac OS X it’s easy to add the DNS entry to the /etc/hosts file, under the 127.0.0.1 entry. This way, your DNS name will always point to your mac, and you’ll be able to reach your virtual host.

I wanted to do the same from inside Windows running on Parallels. An easy way would be to edit the Windows hosts file, adding the Mac OS X public IP to the file. But that will only work if the OS X IP doesn’t change. My Intel mac is an MBP, and I change the network I use often, so I needed a little more flexibility. So, this is the way I found to do this:

1. Configure your VM to use Shared Networking. This wall, Parallels extensions installed on your Mac will create a NAT network where your virtual machine will be hooked into.
   
   
2. Open Mac System Preferences, and look for the “Parallels NAT” network port. This is an interesting one, because it allows the Mac itself to be connected to the virtual NAT network, using an IP on the NAT subnet. Write down that IP: this will be the IP you’ll use to access the Mac virtual hosts from within the virtual machines.
   
   
3. Finally, edit Windows hosts file. This file is located on \WINDOWS\system32\drivers\etc\hosts. Add a line with the IP (in my case, 10.211.55.2) and the name of the virtual host, just like you do on the Mac.
   
   

That’s it. Now you can access your Mac virtual hosts from Windows, whatever the Mac IP is. Ick, what’s a Windows screenshot doing in my blog!?

A not very fast but handy way to downscale images in a WebObjects application is using Java 2D APIs, with code like this:


BufferedImage reducedImage = new BufferedImage(newX, newY, BufferedImage.TYPE_INT_RGB);
Graphics2D g = reducedImage.createGraphics();
g.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BICUBIC);
g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
g.setRenderingHint(RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_SPEED);

g.drawImage(originalImage, 0, 0, newX, newY, 0, 0, originalImage.getWidth(), originalImage.getHeight(), null);
g.dispose();


Although not blazing fast, this is enough for many applications. I could reduce a 7 Mega-Pixels image to something like 250 pixels wide in about one second, or less, in my PowerBook G4. But this was in Tiger.

In Leopard, as some of you may have noticed (and if you have applications deployed on Leopard Server, be aware) this is incredibly slow. When I say slow, I say five minutes, or even more, with the CPU being used at 100% during that time.

There are two reasons that lead to this. The first (which is not a problem in itself, but it’s a cause of the problem): Apple switched from Quartz to Sun2D graphics engine as the default one for Java applications on Leopard. So, all your image manipulation is being done using the Sun pipeline now. This would not be a problem, except for the second reason: the Apple JVM implementation has a bug that is slowing Sun’s pipeline drawImage method to a crawl. Actually, that was not the real reason. I testes this on FreeBSD (using Diablo JDK) and the speed was similar to Leopard’s. Sun2D is REALLY slow, to the point of being useless. I’m now using ImageMagick.

The only solution for now is forcing the application to use Quartz engine. You can do that using the command line option -Dapple.awt.graphics.UseQuartz=true. And, of course, file a bug on this!

Yesterday I spent all day in the first ever Apple developer event in Portugal. Apple carried out a Leopard Tech Talk in Lisbon, where portuguese developers could learn about some of the new stuff in Leopard, including 64 bit programming and Core Animation. The speakers were splendid, and with great technical knowledge about what they were talking about. It’s always great to watch a technical presentation made by real coders, and not by the full-of-bullshit marketing people.

Some presentations were very superficial, but the most interesting ones went as deep as some of the WWDC sessions I attended. The event was actually a micro-WWDC, and even included a nice buffet with plenty of food for lunch and coffe-breaks, all for free. As I spend almost all the WWDC week on the IT track, it was cool to learn about the desktop stuff Apple is working on.

The room was packed, and people were motivated and participative. The Apple guys really liked that, as they say those are the main factors they use to evaluate how successful and event it, specially when going to a new country for the first time. I know I’ll be there next time!

I wrote some days ago about badblocks for testing a hard drive surface. Now, the same for memory.

As I said, I bought a second-hand PowerMac G5 to replace my old G4. When I got the new machine, I run Apple Hardware Test (AHT), using the Extended Test. AHT tested my hardware, including the 2.5 GB of RAM, taking more than two hours (and making a hell of a noise, because during tests, the G5 ventilation system works in failsafe mode, which means, full power). Everything seemed to be fine. Until I installed Retrospect. I use Retrospect to make all my backups at home, and despite all it’s quirks, it always worked fine on the G4. Since I installed it on the G5, I got strange errors (the famous “internal consistentcy check”) and even crashes.

After nailing down all the possibilities (trashing preferences and existing backup sets, reinstalling Retrospect, etc) I suspected it could be an hardware problem, because I was told that the “internal consistentcy check” appears when the backup set contents are corrupted. So, I thought, my hard drive is corrupting data. I duplicated one backup set with about 80 GB, and surprise - after duplicating and running an md5 checksum on it (and diff), the files were different! This was NOT supposed to happen, naturally. So I tried the same thing on my boot drive - same problem. Ops… it’s not the drives. So, if it’s not the drives, and supposing (more exactly, praying) that it was not a motherboard issue, it must be the memory.

All my collegues at IST System Adminstration team use memtest on PCs to test the memory. This great distribution of memtest has a really nice touch: you can burn this on a CD, and boot the PC from it. It takes less that 200K of RAM, so all the other memory will be tested. Unfortunately, it’s not possible to boot it in Macs (not even Intel Macs - I tried it!). So, you must get the Mac OS X version of memtest, and boot the OS in Single User mode (using command-S during the boot sequence). The OS will take about 50 MB of RAM, which is, of course, much worse than the 200K used by the PC version, because those 50 MB will simply not be tested. But it’s better than nothing.

The official site for the Mac OS X version of memtest is here, but unfortunately, the author requires you to pay a small ammount for the download. I don’t like the approach very much because I don’t really know what I’m buying. the author says that, after paying, he sends a password for the encrypted DMG you downloaded. But I cannot download without paying, because the link is no-where. So… what happens when a new version comes out? Do I have to pay it again? Well, anyway, someone else is distributing memtest for OS X for free. Yes, it’s legal, because the software is under GNU license. So, if you don’t want to pay, just click here and grap your own free copy. Happy testing!

By the way, some tests take a lot of time. Let all of them run. Don’t assume the fact that all the “quick” tests passed means your memory is OK. Some problems may only be found with the more complex and slower tests - that’s why they are there. So, let it run. And if you have a G5, get the hell out of there, or use ear-plugs. It won’t be a nice office to work during testing, trust me.

memtest will detetct lots of common problems in memories, and will probably identify more than 99% of the defective memory modules arround. But never forget: it’s impossible to be entirely sure that a memory module is OK, simply because it’s not possible, in a reasonable time frame, to test all the possible combinations of data. Also, memory may pass all the tests in a day, and fail the next day. There are many factors that may trigger a hidden problem in memory modules: temperature, electrical flutuations, the data it contains, age, etc. If you suspect you have a bad memory module, and if you have time, run memtest for several days in a row, using the option to do many passes.

For those of you who need an ethernet card that works with Mac OS X, this may be a useful tip: Mac OS X has a built-in driver for the RealTek RTL8139 chip. I looked arround and found this Netgear card, based on that chip. It’s not Gigabit, but I wanted it to connect my “new” PowerMac G5 to the ADSL modem, so 100 Mbps is fine. It’s not PCI-X, but it complies to the PCI 2.x specification, which means it will work on the G5 PCI-X slots (although the entire PCI bus will work at PCI speed, not PCI-X). It’s cheap, it works, and I can have the G5 doing all the NAT routing stuff, as I like.

There are not many things I miss from Mac OS 9. But there’s one that was really useful: the ability to test a hard drive surface. OS 9 disk formatter (I don’t even recall it’s name) had a “Test Disk” option that would perform a surface scan of the selected hard drive. That was awsome to test for bad blocks on the drives.

Unfortunately, that’s impossible to do with Mac OS X, at least with it’s built-in software. There are some commerical applications to do that (like TechTool Pro), but I get a little pissed off when I have to spend a lot of money buying a software that does a zillion things when all I want is surface scans, and specially when I could do it with the “old” OS and not with the new powerful UNIX-based one.

Well, Linux has the badblocks command that will do just that: test the disk surface for bad blocks. It’s a simple UNIX command, so I thought there must be a port of that to OS X (and, of course, I could try to compile it in OS X as last resource). After some googling, I found out badblocks is part of the ext2fs tools. And, fortunately, Brian Bergstrand has already done the port to OS X, including a nice installer.

The installer installs all the ext2fs stuff, including an extension that will allow you to access ext2fs volumes on OS X. As always, this is a somewhat risky operation. Personally, I avoid as many extensions as I can, because they run too close to the kernel for me to feel confortable. So, if possible, install it on a secondary OS (like an utility/recover system on an exteral hard drive, or so).

The badblocks command will be installed in /usr/local/sbin/badblocks, and it will probably not be on your PATH, so you have to type the entire path when using, or edit your PATH environment variable.

Usage is simple. First, run the “mount” command, so that you know the device names for the drives you want to test. You can obtain something like this:

arroz% mount
/dev/disk0s3 on / (local, journaled)
devfs on /dev (local)
fdesc on /dev (union)
on /.vol
automount -nsl [142] on /Network (automounted)
automount -fstab [168] on /automount/Servers (automounted)
automount -static [168] on /automount/static (automounted)


The internal hard drive is /dev/disk0 (note that /dev/disk0 is the entire drive, /dev/disk0s3 is a single partition). Imagining you want to test the internal hard drive you would type the command (as root):

badblocks -v /dev/disk0

This would start a read-only test on the entire volume. The -v is the typical verbose setting, so you may follow what’s happening. This will take a long time, depending on the hard drive you use. For a 160 GB hard drive, it took between 2 and 3 hours in a G5 Dual 2 Ghz.

I mention this because time is an important factor when testing hard drives! You should run badblocks on a known-to-be-in-good-condition hard drive, so that you can get the feeling of how fast (or slow) badblocks is. Later, if you test a possibly failing hard drive, and badblocks progresses notably slower, it will probably mean that the hard drive is in bad condition (even if it doesn’t have badblocks).

After running the command, you may get two results: your disk has, or hasn’t badblocks! You will see many outputs of a successful surface scan, so I leave here an example of a not-so-successful one:

/usr/local/sbin arroz$ sudo ./badblocks -v /dev/disk0
Password:
Checking blocks 0 to 156290904
Checking for bad blocks (read-only test): 120761344/156290904
120762872/156290904
120762874/156290904
done
Pass completed, 3 bad blocks found.


This is the result of a test on a 160 GB hard drive with 3 bad blocks.

After getting something like this, you may try to run badblocks again, in write mode. Note that this will destroy all the information you have on the hard drive! badblocks won’t copy the information to memory, and than back to disk. It simple destroys it. The point of running a write-enabled badblocks check is forcing the hard drive to remap the damaged sectors. Hard drives have a reserved space to use when bad blocks are found. The bad blocks are remapped to that reserved space, until it fills. And this will only happen on a write. So, run badblocks in write mode, and then again in read-only mode. If badblocks finds no bad blocks, your hard drive is fine (for now). If badblocks still finds bad blocks, it means that there are so many damaged blocks on the disk surface that the reserved area is full. Forget it, and throw the disk away. It’s useless.

I use Retrospect to backup most of the machines at GAEL. You may wonder why do I use a commercial tool that still shows it’s OS 9 roots, instead of open source alternatives. Well, Retrospect has some cool advantages (namely the very good support of laptops that may be disconnected abruptely from the network while a backup is in progress). Also, when I first did this setup, Amanda and other tools did not work reliably with Mac OS X file format.

While this works to backup all the desktop workstations and laptops of GAEL members, I have a problem with our xServe. It runs Mac OS X Server, and Retrospect will not backup machines with the Server version of Mac OS X with the license we have. To do that, we would have to buy a much more expensive license.

No problem. A server, due to it’s nature, doesn’t have the “sudden disappearing” problem of the laptops, so I can use a “classic” UNIX approach - and my choice was rsync and the –link-dest option. You may read about this option in the rsync manpage, but in case you don’t know, what it does is the following: instead of synchronizing a directory in the usual way, it will create a new directory with a new file tree. But, to save space, it won’t copy the non-updated files from the old tree to the new one. Instead, it creates hard links, so that both entries in the file system point to the same data on the hard drive (to the same inode), thus saving space. So, everytime you update your backup, you will create a new tree, but you will only waste the space required by the files that were updated since the last backup, and some more space for the filesystem structures that support the directory tree. You can use a command like this:


// rotate old dirs
rm -rf /Volumes/Storage/test/test.5
mv /Volumes/Storage/test/test.4 /Volumes/Storage/test/test.5
mv /Volumes/Storage/test/test.3 /Volumes/Storage/test/test.4
mv /Volumes/Storage/test/test.2 /Volumes/Storage/test/test.3
mv /Volumes/Storage/test/test.1 /Volumes/Storage/test/test.2
mv /Volumes/Storage/test/test.0 /Volumes/Storage/test/test.1

/usr/bin/rsync --rsync-path=/usr/bin/rsync -az -E -e ssh --exclude=/dev/\* --exclude=/private/tmp/\* --exclude=/Network/\* --exclude=/Volumes/\* --exclude=/private/var/run/\* --exclude=/afs/\* --exclude=/automount/\* --exclude=/.Spotlight-V100/\* --link-dest="/Volumes/Storage/test/test.1" "root@my.machine.com:/Users/arroz/TestDirectory" "/Volumes/Storage/test/test.0/"


Side note: the -E option (capital E) is an option present on Mac OS X rsync version, that forces rsync to copy all the extended Mac file system attributes, including resource forks. It only exists in Mac OS X 10.4 (Tiger) or newer versions. If you are still using 10.3 (Panther) or older, use rsyncx. Do not use rsyncx with Tiger.

Until about a week ago, my backup machine (an old PowerMac G4) had an external SCSI Raid with 640 GB, and an internal RAID 0 (2 * 80 GB drives), besides the boot disk. All the Retrospect backups were being placed on the external RAID, and the server backups were going to the internal RAID 0. Now, I know it’s living on the edge to backup to a RAID 0. But there was really no more space, and it was a temporary situation, because the new drives for the external RAID were already ordered.

When the new drives arrived, I stored all the backups where I could for some days (640 GB was huge when we purchased the RAID, but today is relatively managable), switched the drives and created a new fresh RAID 5. Formatted it in the HFS+ file system, and copied back all the backups, including the server backups and finally trashed the internal RAID 0.

Some path adjustements on my server backup scripts, and we are back in business. But the RAID free space was getting dramatically shorter every day. I used the ‘ls -i’ command to compare the inodes of files that were supposed to be unchanged from the backup of a day to the other in the next day, and as I suspected, rsync was duplicating all the files, instead of hard-linking them.

After Googling a lot, I could not find answers for this. I tried to see if ‘cp -la’ would successfully create hard links, but to my surprise, I found out that the Mac OS X built-in ‘cp’ command would not support the “l” option. Nice. Before installing the GNU ‘cp’ version (and because I’m lazy and I didn’t want to do that) I started thinking about everything I had done since the new drives arrived. The OS was the same, the rsync command was the same, it worked before, so it had to work now. The only reason why it could be not working was because rsync, somehow, thought that all the files were changing, even when they did not.

Suddenly, the solution poped up in my head. Mac OS X has an option, associated to every HFS+ volume, called “Ignore ownership on this volume”. This is turned off by default on the boot drive, but it’s turned on by default on all the external drives you format. There’s a good reason for this: Mac OS X is a consumer product. And average users want to buy an external drive, store data on it, bring it to another Mac, and read their data. They don’t care if their UID is the same on both machines or not.

But this causes serious problems to rsync. Althought the file system will store the owner of the files, it probably won’t report it to the applications who try to read it (or will mask them to the user who’s trying to access them). Somewhere this information is filtered, between the file system and application layers. So, rsync was not getting the real UID of the files. As the files that came from the server had real UIDs, both UIDs wouldn’t match, and rsync would create a new copy because, from it’s point of view, the file had been changed.

The solution was simple - just going to the machine console, and “Get Info” of the external volume. I turned off the “Ignore ownership on this volume” setting, and rsync started operating normally again.