In this post I’ll explain how to recover from a catastrophic failure, like when a hard drive dies. Although much of the steps are identical, this post is focused on how to recover the entire file system and not a small set of files that an user accidentally deleted. To recover something like that, you can simply run amrecover on the server, recover the files you need and transfer them using SFTP or any other protocol to the client machine.

Before going any further, let me just point out an important detail: if you have a downtime on a machine that lasts for a few days, and your tape rotation cycle is not long enough, you may loose your backups at some point (assuming you backup frequently enough). To avoid this, you have several options. The easiest one is to simply disable whatever mechanism triggers your backups (launchd, crontab, etc). Of course, this only makes sense if you have only one machine being backed up on a given Amanda configuration. If stopping the backups is not possible, you can copy the tape directories or, better yet, the tar files that match the affected machine’s volumes to a safe place. Remember you have to copy the correct files, from level 0 to whatever level you are on. If Amanda overwrites the tapes after doing this, you can always recover directly from the tar files.

To recover from an Amanda backup, you should be prepared with a small hardware setup that will make your life easier. I recommend you prepare the following before a problem occurs, so you can avoid doing this under stress. What you’ll need is an external hard drive with Mac OS X installed and the Amanda client installed on it. This way, you can boot the Mac with the freshly installed empty drive and restore the backed-up contents directly to that drive. Note that if you handle PowerPC and Intel Macs, you’ll need to be ready for both situations. The easiest way is to have two drives, although I believe it’s possible to have two bootable partitions on the same drive using a few tricks. Also, I recommend you install the latest OS for both processor families (Leopard on PowerPC, Snow Leopard on Intel).

When recovering, the Amanda client needs to contact the server, which means (assuming you’re using ssh security) you need to generate a public/private key pair on the client and install the public key on your server. I recommend you remove the public key from the server as soon as your recovery operation ends, for security reasons.

So, let’s assume you have a client made out of a Mac with a new, formatted and empty internal hard drive, and running Mac OS X installed from an external drive, with Amanda client installed and a public/private ssh key pair whose public key is already on the server. Let’s also assume the volume list of this machine is the one I recommended on my previous post:

/
/Applications
/Library
/Users

Here are the steps to recover everything from the backups to the empty drive:

1. Launch amrecover on the client using the following command. This will start the recovering console.

/usr/local/sbin/amrecover -C configname -s my.server.com -t my.server.com -oauth="ssh"

Replace configname by the name of the Amanda configuration you want to use for recovering, and my.server.com for your backup server. Note that you must refer to your backup server using a canonical name, not an IP. If you don’t have a canonical name, you can edit /etc/hosts and create one.

2. Run the sethost command to set the host you want to recover. You can get a list of hosts managed by the current Amanda configuration using listhost. To recover from a host called andromeda, you would use:

sethost andromeda

3. Use the setdisk command to define what volume you want to recover. Let’s start with the root volume, so type the following:

setdisk /

Like in the hosts case, you can use the disklist command to list the backed-up volumes.

4. Use the lcd and lpwd commands to move to the root of the empty disk. This will tell Amanda where you want to recover stuff to. If your empty disk is called NewDisk, the command would be:

lcd /Volumes/NewDisk/

You can use the lpwd command to verify you defined the local directory to the correct path. It’s extremely important you don’t forget to define the local directory! Doing so results in Amanda recovering the data to whatever local directory is starts on, which is almost certainly not what you want to happen.

5. Add the list of items to recover.

We want to recover everything, so let’s just add the root directory:

add /

You can use the usual UNIX commands (cd, pwd) to navigate trough the backed-up hierarchy and select exactly what you want to recover. Note the paths are relative to the root of the current volume, not the root of the original client.

6. Initiate the recovering procedure. To do that, simply use the extract command:

extract

Amanda asks you to confirm the operation, and will ask you if you want to load the needed tapes during operation. For unattended operation, I recommend you to hit the return key several times to answer “Yes” to all the tape change questions (assuming you’re using virtual tapes, and not real tapes, of course). This will take a while.

Repeat the following 3 steps for the remaining volumes (I’ll use /Library as an example):

7. Use Finder or another terminal window to create the root folder of the volume:

cd /Volumes/NewDisk
mkdir Library

8. Back on Amanda, change disk and local directory to match the new volume:

setdisk /Library
lcd /Volumes/NewDisk/Library

9. Add all the files in the current volume, and initiate extraction:

add /
extract

Again, repeat the previous steps for all the volumes you want to recover.

10. Use Disk Utility to Repair Disk Permissions on the new disk. This will make sure the proper ownership and permission settings are assigned to the directories you created (and everything else).

You should now be able to boot from the recovered drive. You’re done. :)

As with any UNIX tool, Amanda can be compiled, installed and configured in a lot of different ways. How you should do it depends on your needs, so don’t feel pressured to do everything in the same way I did, as it’s not “the right way”, just one way. Also, everything I describe here should work on the Leopard or Snow Leopard versions of either Mac OS X or Mac OS X Server on Intel or PowerPC Macs. I’m not sure about previous versions of the OS, but you may find more information about those in the Mac OS X installation notes page of the Amanda wiki.

Also, I won’t write anything here about installing an Amanda server. I’m assuming either you already know how to do it and just want to check how to backup Mac OS X clients, or that you are new to Amanda and will follow the appropriate tutorials and do some experimentation before moving Amanda to production.

Before installing Amanda on a client, you need to decide what you want to backup, and how to split that into Amanda “disks”. I usually backup the entire machine (a single Mac OS X volume), so the obvious choice to make would be to consider the root of the file system as the only Amanda disk. But I recommend you to not do that. Why? Two reasons:

• If you consider the entire Mac OS X volume to be a single disk, a level 0 backup will use a lot of space, and take a long time to complete. Even worse, if the space needed for a level 0 backup is as large, or larger, than a single tape, you may not be able to perform the backup at all. By splitting the volume into smaller disks, Amanda will spread level 0 backups of those disks over time, which will also spread the space and time used for those necessary level 0 backups.
• Different parts of your Mac OS X volume have different usage patterns. The /System directory, for instance, contains mostly read-only files that change only when installing a system update, which is a relatively rare event. On the other hand, the /Users folder contains user home directories, whose contents may change on a daily basis. This different behaviors lead Amanda to optimize it’s planning, specifically the decision to maintain of increase the backup level of a given disk, as explained in the previous article. If you don’t split two main directories like these, with radically different usage patterns, in different disks, you are making it harder for Amanda to make the right decision.

The way I split an OS X volume into several disks depends on what machine is used for. However, I end up doing it always the same way for normal use desktop machines. I create the following disks:

/
/Applications
/Library
/Users

This usually works fine. Note that, if you are not careful, by considering / (the root) a disk, you’ll include all the other disks (those directories are inside the root) and any other devices (hard drives, CD-ROMs, USB pens, etc) that may be plugged in that computer (they will be mounted into the /Volumes directory). I was told Amanda won’t include mountpoints (which means it would not backup anything inside /Volumes anyway) but I haven’t tested. We’ll see how to solve this issue shortly.

If your users home dirs contain a lot of data, or data that don’t need to be backed up, you may consider splitting the /Users directory into smaller disks, and excluding easily restorable directories, like ~/Music. I never had to split the /Users directory, but I do exclude the ~/Music directories and any directories that contain virtual machine hard drive images (like Parallels or VMWare). Those files are huge, and change every time an user launches their virtualization software. If your users don’t have important data on those virtual hard drives (like in my case, we only use Windows to run IE), you can easily re-install Windows from scratch. If you do have important data on those drives, I recommend you install backup software on the virtualized OS to keep that data safe.

Ok, enough talking, let’s do it.

1. Install a recent version of Xcode. You’ll need it to compile the needed stuff. Xcode 3.2.3 is the most recent version at the time I’m writing this and will do just fine.

2. Install MacPorts. Amanda needs glib2 to compile. glib2 has a lot of dependencies, so the easiest way to install it is using MacPorts. To do that, simply open the MacPorts installation page, download and run the appropriate installer for your version of OS X.

3. Install glib2. After having installed MacPorts, this can be achieved using one simple command (run it as superuser):

/opt/local/bin/port install glib2

All glib2 dependencies (and glib2 itself) will be downloaded, compiled and installed. This will take a while, specially on PowerPC Macs, so if you need to have a snack or refill your coffee mug, this is a good time to do it.

4. Create a user for Amanda and add it to the admin group. The best way to achieve this is to run the following list of commands as superuser:

sudo dscl localhost -create /Local/Default/Users/amandabackup
sudo dscl localhost -create /Local/Default/Users/amandabackup RecordName amandabackup
sudo dscl localhost -create /Local/Default/Users/amandabackup UserShell /bin/bash
sudo dscl localhost -create /Local/Default/Users/amandabackup RealName "Backup User"
sudo dscl localhost -create /Local/Default/Users/amandabackup UniqueID 5000
sudo dscl localhost -create /Local/Default/Users/amandabackup PrimaryGroupID 0
sudo dscl localhost -append /Local/Default/Groups/admin GroupMembership amandabackup
sudo dscl localhost -create /Local/Default/Users/amandabackup NFSHomeDirectory /Users/amandabackup
sudo ditto -rsrcFork '/System/Library/User Template/English.lproj/' /Users/amandabackup
sudo sh -c "echo 'amandabackup_server.example.com amandabackup' > /Users/amandabackup/.amandahosts"
sudo chmod 600 /Users/amandabackup/.amandahosts
sudo chown -R amandabackup:wheel /Users/amandabackup
sudo passwd amandabackup

The last command defines a password for amandabackup user. You can later configure your ssh deamon to accept only public key authentication.

5. Download Amanda. You may obtain Amanda’s source code from the Amanda downloads page. Although the 3.* versions are already available, I’m using 2.6.1p2 for now. This instructions assume you’ll be using the same version. I’m not sure if version 3 compiles well on OS X. After downloading, uncompress the file.

6. Compile and install Amanda. Amanda is installed using the standard UNIX configure/make/make install commands. You have, however, to provide some information on the configure command. The one I use is:

./configure --with-user=amandabackup --with-group=admin --with-ssh-security --without-server

After that, just run the usual:

make
sudo make install

7. Create the /usr/local/var/amanda directory with the appropriate ownership. The install procedure somehow fails to create this directory, so you need to create it manually and give it the right ownership. So, as superuser, run the following commands:

mkdir -p /usr/local/var/amanda
chown amandabackup /usr/local/var/amanda

Amanda is now installed on the client. If you are going to use ssh as the tunnel for your connection, you should now copy the server’s ssh public key to the client and test the connection. You can then add the client’s volumes to the disklist and run amcheck to test the configuration.

Now you need to add exclude files to make sure the root volume won’t contain all the others, and to exclude large files whose backup is not needed (like music or movies). To do that, you have to add a configuration directive in your Amanda server configuration files. That directive tells Amanda the name of the file that includes the exclude list, which is the list of files to be excluded from backup. This is a sample configuration I use:

define dumptype comp-tar {
    program "GNUTAR"
    compress fast
    index yes
    record yes
    auth "ssh"
    exclude list optional ".amanda-exclude.list"
}

Now you need to create the exclude files where needed, named “.amanda-exclude.list”. I usually use two of those files. One in the root of the file system with the following content:

./Volumes
./Applications
./Library
./Users
./home

This excludes the other Mac volumes (that are mounted in the /Volumes directory) and the other defined Amanda volumes (the word “volume” is used on both contexts which may lead to confusion, but you want to exclude both kinds of volumes, so it’s ok). If you’re wondering about /home, it has no useful content (I suppose it’s there just for compatibility with some UNIX tools) and causes problems if not excluded.

The second file would go in /Users directory, and contain something like this:

./<user>/Music
./<user>/Documents/Virtual Machines.localized

Replace <user> with the username of the machine user, and duplicate the lines for additional users. This would exclude the user’s music directory and the VMWare disk images. Of course, your milage may vary. If you want to backup everything, just don’t add this second file.

You should now have a perfectly functional Amanda client.

I’ve been using Amanda for a while now to backup all the Macs in our workgroup (10 machines) and so far I’m nothing but happy. Amanda is an open source backup system for UNIX-based operating systems, Mac OS X included (I believe it can also backup Windows clients, but I couldn’t care less).

Amanda was originally designed to backup to tapes. Today, since hard drives became cheap and are a great media for backup, Amanda also supports virtual tapes. A virtual tape is a directory on disk that essentially acts as a tape, storing raw information. While configuring an Amanda server, you create a set of tapes of an arbitrary size (100 GB, for instance) and Amanda will use at least one tape per day, eventually rotating trough all of them.

I’ll now lay down some considerations about how Amanda works and how does that reflect on usage and backup planning. Also, I’ll do some comparisons with Time Machine.

Scheduling

In amanda, you define one or more configuration files. Each configuration can have a set of hosts (a backed-up machine), and for each host a set of disks (a disk is any directory you want to backup, it’s not needed to match Amanda disks with physical disks or logical HFS+ volumes). You generally run a backup operation that will backup all the disks of all the hosts on that configuration file.

Launching a backup operation is done by simply executing an UNIX command from whatever mechanism you prefer (cron, launchd, manually, etc). This means you can define how often does your backup run, and at what time. Usually, a daily backup is performed, but you may want to backup every 6 hours or only once per week, depending on how often does your data change and how bad is it to loose a few hours of work. This means that, in Amanda, the server is proactive and initiates the backup procedure whenever you set it to. In Time Machine, backup operations are initiated by the clients every hour. The server is simply a dumb file server where backups are stored.

Storage

Amanda uses the tar format to store data on a virtual tape. For each backed-up disk Amanda creates a tar file inside the tape with data. Amanda stores data in incremental fashion, using some redundancy as well. This is implemented trough the concept of backup levels. The base backup (where all the files on a disk are backed-up) is level 0. From there, backup levels are incremented, and each level essentially means it contains the incremental changes relative to an archive in the immediate lower level. This means a level 3 backup contains the changes relative to a level 2 backup. That level 2 backup contains the changes relative to a level 1 backup which contains, as expected, the changes relative to the level 0 backup.

Amanda decides on the level is should operate on based on a complex planner that considers several factors that result in a decision. You may add some configuration options to customize the weight of some of those factors in the final planner decision. Essentially, Amanda tries to obtain the right balance between reliability (which, in this context, means the probability of recovering a backup successfully) and used disk space. Reliability decreases as the backup level increases, due to the fact that, if you want to recover a level 5 backup, you need to have the consistent level 0, 1, 2, 3, 4 and obviously 5 backups, because each of those build upon the previous one. A level 0 backup is more reliable in the sense that you only need the level 0 backup itself to recover. Of course, a level 0 backup is an “everything” backup. If you do a level 0 backup every day, you’ll use a lot of disk space.

This is an example of a production Amanda installation for a host called bergman and volume / (the root):

date                host    disk lv tape or file file part status
2010-07-01 06:05:24 bergman /     2 DAILYS-6       15  1/1 OK
2010-07-02 04:19:11 bergman /     3 DAILYS-7       23  1/1 OK
2010-07-04 04:47:44 bergman /     3 DAILYS-9        9  1/1 OK
2010-07-05 02:51:01 bergman /     0 DAILYS-10      21  1/1 OK
2010-07-06 02:51:34 bergman /     1 DAILYS-11      11  1/1 OK
2010-07-07 04:59:40 bergman /     1 DAILYS-12      12  1/1 OK
2010-07-08 02:59:32 bergman /     1 DAILYS-13      15  1/1 PARTIAL
2010-07-10 02:46:41 bergman /     1 DAILYS-15      20  1/1 OK
2010-07-11 05:25:49 bergman /     2 DAILYS-16       8  1/1 OK
2010-07-12 03:43:04 bergman /     0 DAILYS-17      16  1/1 OK
2010-07-13 04:25:23 bergman /     1 DAILYS-18      11  1/1 OK
2010-07-14 02:56:48 bergman /     1 DAILYS-19      15  1/1 OK
2010-07-15 02:54:01 bergman /     2 DAILYS-20      11  1/1 OK
2010-07-16 02:46:30 bergman /     2 DAILYS-1       11  1/1 OK
2010-07-17 05:05:08 bergman /     3 DAILYS-2       12  1/1 OK
2010-07-18 03:00:43 bergman /     3 DAILYS-3        9  1/1 OK
2010-07-19 05:11:36 bergman /     3 DAILYS-4        7  1/1 OK
2010-07-20 02:49:03 bergman /     4 DAILYS-5       24  1/1 OK

The backup level is indicated by the “lv” column. As you can see, I have 20 virtual tapes, and the last one to be used was DAILYS-5. The next one will be DAILYS-6 (its content will be erased and the tape will be reused). You may be asking yourself why are there two level 0 backups, and a lot of consecutive repeated backups with the same level.

Consider two rules of thumb to understand that:

1) A level 0 backup is needed to recover, so Amanda must make sure at least one level 0 backup exists at any time, given the number of tapes and their rotation. Also, bad stuff can happen, like a machine being down or unavailable at the time the backup runs. If that happens during a few Amanda runs, it may happen than you hit a number of rotations where you effectively loose the level 0 backup without creating a new one. That’s why Amanda makes a few level 0 backups among the way, to make sure you still have a second level 0 backup if the first one is destroyed. You can configure the maximum number of runs that go by without a level 0 backup being created. It’s highly recommended that that number is lower than the half of the total amount of tapes you have, so that, in the worst case (like in my example, where I have 20 tapes), you have a level 0 backup in the “middle” of your tape recycling circuit. Now, why did Amanda created two level 0 backups, one in July 5, and another one in July 12, effectively less than the maximum time allowed? See below.

2) Despite what seems intuitive, Amanda does not increase the backup level every run until it gets back to zero again. I don’t know in detail all the data the planner uses to make a decision, but there are at least two interesting considerations the planner takes into account.

The first one, is a level N+1 backup a lot smaller than a level N? If the answer is no, Amanda decides to keep the same level. There’s no point in increasing the level (and lowering reliability) if no significative amount of disk space will be saved. This may happen if you change approximately the same set of files in each run. In that case, the incremental backup from level N to N+1 would be of almost the same size as the N-1 to N.

Second, Amanda sometimes promotes level 0 backups ahead of schedule to spread them trough time. Doing all the level 0 backups in the same day would be very slow and might not fit in the maximum number of tapes allowed for a single run (you can define that value on the config file). There’s also another factor: free space on the tape. Remember that, for now, Amanda cannot use the same tape in two consecutive runs, so there’s no point in leaving unused space on a tape. If there’s enough space to perform a level 0 backup instead of a higher level, Amanda may decide to do it.

As you can see by now, data storage is handled quite differently in Amanda and Time Machine. I’ll assume you know how Time Machine works, so I’ll get straight to the point: in Amanda you may have more than one copy of the entire volumes you are backing up (ie, several level 0 backups of the same data), so you need to take that in consideration while planning storage space. On the other hand, amanda allows gzip compression (and encryption) of backup data, so it’s not completely obvious how much more (or less) space you need for Amanda backups compared to Time Machine. In some situations, if your data is highly compressible, you may even end up with two level 0 backups taking less space than a single Time Machine backup, although the opposite will happen most of the time. What’s cool is that if you use compression, Amanda learns with time how compressible your data is, and adjusts it’s planning according to that.

Another interesting note is Amanda using standard file formats for storing backups (tar and optionally gzip). This allows recovering of data even in machines where Amanda is not present. If you navigate into a virtual tape directory on your file system and run the “head” command in one of the stored files, you’ll see something like this:

AMANDA: FILE 20100715010001 serpa /Library  lev 2 comp .gz
  program /usr/bin/gnutar
DLE=<<ENDDLE
<dle>
  <program>GNUTAR</program>
  <disk>/Library</disk>
  <level>2</level>
  <auth>ssh</auth>
  <compress>FAST</compress>
  <record>YES</record>
  <index>YES</index>
  <exclude>
    <list>.amanda-exclude.list</list>
    <optional>YES</optional>
  </exclude>
</dle>
ENDDLE
To restore, position tape at start of file and run:
	dd if=<tape> bs=32k skip=1 | /usr/bin/gzip -dc |
          /usr/bin/gnutar -xpGf - ...

The first part is metadata used by Amanda. After the metadata ends, there are two lines used to tell you how to restore the data stored on that file with standard UNIX tools. This means you don’t need to worry about being able to recover your backups if Amanda development happens to stop for some reason. As long as you have an UNIX machine, you’ll be able to restore your data.

Security

There are two points I want to mention about security: transport and storage encryption, and system architecture. I’ll assume we’re always talking about having a backup server and several clients. If the problem you’re trying to solve is so simple that it can be fixed with an USB disk and Time Machine, you’re just wasting your time reading this. :)

Amanda supports encryption both during the data transport and on data storage. Transport security is guaranteed by using ssh with public/private key pairs. It also supports data encryption on the data storage by using symmetric private-key based encryption of public/private key pairs. Encrypting your backups is important, specially if you store them in an offsite location (either via network transfer to a remote data center, or by physically storing hard drives or real tapes in a safe). In the event data ends up in the wrong hands, encryption will rend it useless for the bad guys. Things are substantially different in Time Machine. Data storage encryption is simply non-existing (unless you use a lower level encryption method, like PGP). Transport encryption may be provided by the AFP protocol, used by Time Machine to connect to the backup server, depending on your server configuration.

More interesting, in my opinion, is the implicit security resulting from the client/server architecture used by Amanda. A backup server should be one of the most safer and well guarded machines you have in your network. All your data will be there. If the backup server gets compromised, it means all the data from all the backed-up machines might now be in the wrong hands. You would want your backup server to run as few services as possible, and to allow access to the server (like ssh) only by trusted admins from controlled networks.

In Amanda, this is possible. As I described before, when a backup operation starts, the Amanda server will contact its clients using the method you chose in the configuration (ssh in my case). So the connection is made from the server to the client, and not the opposite. This means your user’s machines (which will naturally be less secure than your server because those pesky users will run all the crap they get from the internets!) will never access the backup server, and though compromise it’s security. Time Machine works in the opposite direction. There’s no concept of a “backup server”. The clients run the show, and the server is simply a file server where backups are stored. This exposes the user backups to whatever malware and trojan horses they may have running on their Macs. If the server is misconfigured, or if some hacker exploits an unknown vulnerability in the AFP protocol, other users backups may be compromised as well.

Conclusions

Amanda may be a great option to backup always-on Macs, like xServes or desktop machines (along with Linux or FreeBSD servers, of course). It offers a vey fast, reliable and secure infra-structure upon you may build your backup system. However, it lacks the user interface and simplicity of Time Machine (most configurations require sysadmin intervention for recovering data from a backup). Also, Time Machine may be more appropriate if you rely heavily on laptops that will not be available on the network on a predictable schedule.

Amanda pros:

• Works on any UNIX system (and Windows clients) which may help when planning a multi-OS backup scenario.
• Offers very good control of used disk space.
• Allows data encryption and compression.
• Secure client/server architecture.
• May be used on real tapes, not just hard drive-based backups.

Amanda cons:

• Not appropriate for laptops that may be out of network reach during backup operations.
• Step learning curve, a little hard for beginners to configure and get everything running smoothly.
• Unless your users are computer experts and have access to the backup server, it requires sysadmin intervention for recovering data.
• Recovering a single file may take some time, because the entire tar archive has to be read until the file is found.
• Backups are usually less frequent than Time Machine.

Time Machine pros:

• Very simple setup.
• Non-expert users may recover lost files and easily browse filesystem history using the stunning user interface, without requiring sysadmin intervention.
• Works fine with laptops with intermittent network connections.
• Very well integrated with Mac OS X, makes backing up and recovering very easy and part of normal usage and new machine installation workflow.

Time Machine cons:

• Works only on Mac OS X, and requires a Mac OS X Server as backup server (unless you go with unsupported devices and face possible consequences).
• Offers no data storage security.
• Very hard/impossible to control data storage strategy and used space.
• Requires clients to access server, which decreases security.
• Works only with disk-based backups.

There’s no clear winner here, it highly depends on your needs and restrictions. I hope this article gave you a general idea of what Amanda is, how does it work, and it’s advantages and weaknesses. In the next article, I’ll describe how to install a Mac OS X amanda client, and how to recover from a catastrophic drive failure.

Retrospect 6 was a very good backup software, but its age was showing up. It was still fundamentally a Mac OS 9 app running on top of OS X Carbon API. The worst part about this was the need to launch the application in Finder, forcing you to have automatic login configured and remote-acess your backup machine via Remote Desktop or VNC, assuming it was running on a data center. Despite that, it was very reliable (I had two situations where Retrospect complained about corrupted data, and both situations were caused by faulty hardware). Also, it was not the fastest software I had seen, but it was good enough. Keep in mind Retrospect was designed when file systems had hundreds or a few thousand files on them, not a million or more like it’s normal today (I have about 1.5 million files on my laptop drive).

Facing the unavoidable, EMC decided to re-write Retrospect from the ground up as a modern OS X product, using Cocoa APIs and changing it’s architecture to a proper UNIX daemon (with a remote graphic console). Besides theoretically solving all the version 6 downfalls, they added some nice goodies, like AES 256 encryption, grooming, and some details. A nice one is that the disk backups are now stored in 100 MB files instead of a unique, giant file. This solves a lot of problems related to NAS systems.

Assuming you need backup software, you would think this were great news, right? Well, so did I. However, reality seems to be a lot worse than the perfect scenery I described above. EMC worked a lot of time on this version, and you would think they had made it, made it good and made it fast, right? Well, they decided to stop somewhere in the “Make it good” part. The problem is, rules of thumbs are nice, but common sense helps. If you launch a product that is simply too slow to be useful, people won’t use it.

Our backup machine at the university is an old G4, dual CPU (1 Ghz, I guess). Yes, it’s not exactly a screamer, but what the hell, we are talking about copying and storing files. It’s not exactly rocket science, and if the machine was up to the task when it was new, it should be up to the task now. We upgraded the RAM to a decent amount, of course.

So, last week I needed to recover the /etc directory of a colleague’s laptop to recover some apache and PHP settings that Snow Leopard installer happily overwrote. We’re talking about 3 MB of data, something that should be as simple as pressing a few buttons and get the data back. The laptop had about 750 thousand files on it, which, in my opinion, is not that many for today’s standards. So, why the hell took me about 3 hours to recover those files? Loading the catalog into the UI took almost 2 hours. Deselecting the whole file tree took almost 1 hour. The rest was the recovery process itself.

Ok, EMC. I know this stuff is optimized for Intel, and you do a lot of byte-order swapping on PowerPC. I know I’m using an old machine. But for god sake. What the hell are you doing to my CPU that needs 3 god damn hours to load a 750 000 files catalog into memory? And what’s the story with deselecting all the files? Are you telling me that, when I press the checkbox on the file root, you REALLY go trough the entire file tree and deselect each individual file? (In a rather inefficient way, because changing 750 000 booleans would take about… what? 1 millisecond on a 1 Ghz CPU?)

Well, I have at home a Dual 2 Ghz PowerPC machine that acts as backup storage, among other things. I have an AES 256 encrypted disk image (note that, at the university, I’m not using encryption, or else it would render things unbearable), served by AFP, acting as the time machine target for my laptop. When I installed Snow Leopard, I also had to recover some stuff from the /etc directory. Do you know how long it took? About 5 freaking seconds! And please, don’t tell me it’s because the G5 is faster!

Enough is enough, and 3 hours to recover a few files is ridiculous. I’ll study the possibility of migrating backups to Time Machine. Yes, space management is worse. Yes, client machines have to access the server, making the whole setup less secure than Retrospect where the server accesses the clients. But people won’t have to stop working for hours waiting for some files to be recovered.

Lesson for you, EMC: I’m not asking for you to make it fast before making it good, but at least make performance acceptable before releasing the product. Specially after taking so many years to build it.

• Please fix the wireless driver that causes my Mac to crash about 10% of the times I turn Airport off.
• Please fix the trackpad driver, or whatever is causing the trackpad to behave strongly erratic during about 30 seconds after waking the Mac up.
• Please fix the damn copy/paste bug that makes the paste command paste the previously copied object and not the most recent one. This is specially irritating when you cut a piece of text, paste and you realize you are pasting something else, and that your supposedly cut piece of text is lost forever, unless you can undo and get it back.
• Please fix the irritating bug that causes an iChat window to keep being the active one even after I click Safari, making its window go in front of iChat’s. That’s specially annoying when I type apple-W to close the Safari window, and the ichat one goes away.
• Please provide replacement keyboards for people who has pre-unibody MacBook Pros that, you know, actually sense a keystroke every time the key goes all the way down, without the need to almost punch the key.
• Please fix whatever is causing my father’s MacBook Pro to keep waking up and going back to sleep when the lid is closed and the charger on, despite I had already turned off every god damn thing that could wake it up, including the lid open event.
• Speaking about the charger, please provide chargers where the charge light doesn’t go off for some unknown reason. It still works, but it doesn’t inspire a lot of confidence in it ans it’s safety.
• Please provide granular updates to Mac OS X Server. Please please please pretty please.
• Please care a little more about the entreprise and IT markets, namely your own web application technology (WebObjects, of course).

Thank you, guys! You must hate me but you’re nice people anyway. Sometimes.

Versions is finally out! :) João Pavão, together with the Sofa team, released their new Subversion client. It’s a really powerful application made by people who deeply understand how a Mac application should be. In a world full of dubious software, it’s good to see that some people still care a lot about their code quality and the detail. I feel honored for having been one of the few who saw this application being born and getting mature enough to be released in the wild. Congratulations, João! :)

As I previously wrote in my blog, I was waiting for a whole disk encryption solution to be made available for Mac OS X. Some months ago, Checkpoint released what I believe was the first solution ever to support full disk encryption on the Mac, including the boot disk. Recently, as I noted before, PGP Corporation release PGP Desktop 9.9 for Mac OS X, supporting full boot-disk encryption for the first time on this platform. I opted for trying PGP, as they made a demo version available (that will work for 30 days) and it’s possible to buy a license online. Checkpoint, on the other hand, doesn’t seem to have a downloadable demo, and doesn’t sell the product online. This was enough for me to forget Checkpoint solution at all, specially having the PGP demo ready to be downloaded from their site and installed. So, PGP it is.

Why should you care?

Today, you easily buy a laptop with a large hard drive. That drive will quickly be full of important data. Many people consider “important data” to be private emails, trip photos, holiday movies, and such. But that’s just the tip of the iceberg. Letting anyone read your email or pick at your photos can be a problem, but it’s nothing compared to really important data. I’m talking about company reports. Source code. Data about your customers. Intelectual property. Financial data. Anything that keeps your business going, and that can put it in a very delicate position if it gets to the wrong hands.

Also, there’s an even more important fact you should take into account: some data you are carrying on your laptop is not yours, but other people’s data. Confidential emails with clients or business partners. Marketing and product information that should not be disclosed before a certain date. Governmental and military information, including private data from citizens (as an example, recently some events like this one happened in UK, where laptops with sensitive official information were stolen or lost). This means that you are no longer responsible just for your data security, but also for other people’s.

On top of this, you must think about the consequences of letting this information be revealed to the wrong persons. Important information about your clients may leak. Intelectual property that keeps your company ahead of the concurrency may become public, destroying your company advantage or, in the worst case, destroying the company itself. Governmental agencies may be placed in the hot seat for letting private information about the citizens be stolen and accessed.

All this together should be more than enough to make you worry about your computer’s data security and convincing you to do something about it.

Whole disk encryption

Computer security is a very wide subject, and there are a lot of things to consider. Network security, host security, etc. In this article, I’m covering whole disk encryption. So, what is whole disk encryption?

Whole disk encryption is a technique where all the drive contents are encrypted using a secret key (which can be a password, a key stored in a USB dongle, etc). When I say “all the drive contents”, I mean it. Even the operating system is encrypted. This means that you will only be able to access that drive’s contents if you have the key to access them, and this includes booting the computer from that drive. If you don’t have the key, you won’t be able to read the data whatever you try. Mounting it on other computers of physically installing it on a different computer won’t work. For anyone who doesn’t have the key, the drive will be as good as an empty one. All the contents will appear to be random garbage.

For those who have the right key, the main advantage of whole disk encryption is that it won’t affect the computer usage at all. The only thing you have to do is to type in the password right after powering up your machine. After you type the password, the OS will boot normally and the machine will work as if nothing special was happening. The secret is that PGP runs between the hardware and the OS itself, intercepting all the data input and output from and to the hard drive. All the applications, and even the OS itself won’t even realize that the hard drive is encrypted because the PGP layer will decrypt data requested by the applications on the fly. This is great, because it makes very unlikely that some application won’t work because of the disk encryption process. All the magic happens below the OS itself, as close to the hardware as it can be.

This will protect your data against one of the attack vectors that is most hard to defend yourself against: someone having physical access to (and some time alone with) your machine. This included the machine being stolen (which is very likely to happen at some point to laptop computers), someone entering your home or office and remove a hard drive from a computer, and even accessing data centers and stealing hard drives or entire servers (and if you think that doesn’t happen… think again, it’s more frequently that what most people believe).

I want to make clear that this will not protect your mac against other types of attacks. As I stated before, the OS and the applications will run in the same way they did before. So, if you have a virus or a trojan horse on your system, the virus or trojan will work. If you have a compromised network service, hackers will be able to get in using it. If you download an application that erases all your files, all the files will be erased. The whole disk encryption system has the only purpose to keep all the data on your hard drive protected when the system is not running. As long as you type in the password and boot the OS, all the OS-level security weaknesses that were there before will be there again. PGP Desktop has some more security features but I won’t cover them here.

What about Apple’s File Vault?

Apple provides you with some “transparent” data encryption features on Mac OS X, namely File Vault. File Vault will encrypt all the files in your home directory and store them on an encrypted disk image. You will always be able to turn the Mac on, but you must provide your account password on the login for that disk image to be accessed. As with PGP, data will be encrypted and decrypted on the fly. So, why not use it? There are many reasons why using File Vault is impractical:

1. It’s not whole disk encryption, only home directory encryption. One may argue that all the important files are in the home directory, but that’s not entirely true. Many applications write temporary files to directories outside of your home directory, like /tmp. This files may contain sensitive information, and that information will be recorded unencrypted on your drive. Also, software like databases of other kind of servers store their data outside of user’s home directories, and that data will also be stored in clear.
2. It conflicts with some applications, specially backup solutions. For any application executed by another user, including the OS itself, a user home directory will be a single, huge file, the encrypted disk image. The backup software will not be able to peek inside your home and only backup the files you changed since the last backup operation, so it will try to copy the entire file. Worse yet, if you change the file during the backup, you can corrupt the backup, making it hard or impossible to restore it if needed.
3. For the same reason, remote services will not work because they won’t be able to decrypt your home directory. This is the case of a remote shell, for instance. If you ssh to a Mac with your home directory under the domain of File Vault, you won’t be able to access your files.
4. It’s slow an unreliable. File Vault works by creating an encrypted virtual file system inside a file that grows and shrinks as needed that is itself stored in the real file system on your drive. There’s a huge load of things that can go wrong with this. This is corroborated by the fact that every time I tried to create disk images with many (hundreds of thousands) of files, the disk image inevitably corrupted and I could not access it’s content any more. Don’t forget that your entire home directory will be really a single huge file with some complex data and mechanisms that make it work. Now compare this with the simplicity of the PGP solution: just insert a layer between the OS and the drive, and don’t ever think about it again. It just works, it doesn’t need to care about files, folders, file systems, or anything else. It’s just raw data. OS asks it to write a sector on the drive, PGP layer encrypts the sector, and the sector goes to the drive. No complex processing, no complex data modeling, no complex code to fail. The PGP layer doesn’t even need to know what’s doing, it blindly encrypts and decrypts data on the fly. The OS will know what to do with that data.

For these reasons, I believe whole disk encryption is a much better solution than File Vault. I strongly believe Apple should provide this with their Macs right out of the box, but judging by the way the company handles security issues, I don’t believe that will happen any time soon.

PGP

What can you say when a product that is supposed to do what it does in the background and be totally transparent to the user actually works fine? Well, nothing. That’s precisely the point – providing security without being a pain to the user. So far, that’s my experience with PGP. I really have nothing much to say, except that it works.

I installed PGP, rebooted and typed in my demonstration registration key, valid for 30 days. Then, I read the manual, skipped all the “please verify your file system consistency before proceeding” warnings (what could go wrong?), set a password for my MacBook Pro drive, and fired up the encryption. You can use your mac normally while the initial encryption is done, as PGP is smart enough to know what disk sectors are already encrypted and which ones are not, allowing the system to work normally during the whole process. You will probably notice a very high loss of performance during the initial encryption process because the hard drive will be in really heavy usage (after all, PGP has to read and rewrite the entire disk surface).

After that, you won’t notice a thing. The only signs your mac will show you related to PGP are the small PGP icon on the menu bar, and, of course, the password window before the system boot. You won’t notice any performance degradation due to the real time encryption, at least I didn’t. Based on the UNIX “top” tool, it appears that PGP doesn’t use more than 2 or 3 percent of the CPU, which is negligible (remember that we are talking about 100% per CPU, which means that in a modern laptop with a Core 2 Duo processor, PGP is using 3%… of 200%).

You can create several “users” for your hard drive, with different passwords for each one. Please keep in mind this is only a way to avoid sharing the passwords. This is NOT a real accounting feature like in a normal UNIX system, where each user has different permissions and credentials to (supposedly) access only what he should. Here, any password will provide access to the entire drive contents. The normal access permissions will be granted by Mac OS X, of course, but PGP offers no data protection as soon as someone – whoever it is – types in a valid password.

Side notes

There are some important things to keep in mind when using PGP to encrypt your drive:

• PGP asks for the password on system boot, but not on system awake. So, make sure to turn on the system awake password in the System Preferences, or you can allow a burglar to access your data if he stoles your Mac while it’s sleeping (and at least I always carry my laptop in sleep mode). If you want absolute security, turn off your mac before taking it with you.
• Target mode (using your mac as a firewire drive) will work, but the drive contents will not be decrypted on the fly by the machine in target mode itself. You have to install PGP on the host machine so that it’s able to decrypt the contents of the target disk.
• You have to be careful if you need to clone an encrypted drive. The most reliable way to do it is to decrypt it before cloning. If you want to clone an encrypted drive, check this thread in the PGP Forum for more details.
• I haven’t tested this, but from what I recall from the manual, you may install PGP on a machine and use it unlicensed to read the contents of an encrypted drive (assuming you know the password, of course).
• BootCamp won’t work. If you need Windows, you have to run Parallels, VMWare or any other virtual machine software. Those will work fine, providing that the Windows disk image is a file on the OS X file system, and not a dedicated Windows-formatted partition.

I hope they release a demo version, as I would like to know how the system reacts when awaking the machine from sleep and using target mode. Until then, you can register yourself at the PGP site to be notified on the release date.

• As David LeBer already mentioned, Pascal did an amazing job organizing WOWODC all by himself. Great room (a bit cold on the first day ;) ), large windows and sunlight on the halls. Food (well… not that good, but after all, it’s USA!) and caffeine provided frequently. Very very nice. Suggestion for the next year: a bigger (and brighter) screen, and eventually plasma screens among the room to make it easy for people in the back to read the code.
• I learned a lot about WO frameworks out there (like Wonder, Houdah and specially LEWOStuff that I did not know before). I met for the first time some very talented people, and of course, all the folks from the previous conferences. It’s great to be able to have technical discussions and know different views on the same problems from all those skilled and experienced people out there, face to face.
• WWDC had some interesting news on many stuff. As you know, I cannot talk about the stuff under NDA, so I shall only say that some interesting stuff is being done on the WO side. Also, as you all know by now, the iPhone is now 3G, includes a GPS, the price was slashed, and will be available in many countries of the world. I just hope the service providers slash the data roaming prices, because that makes the iPhone useless when you go to foreign countries. Finally, Snow Leopard was announced, and, as already expected, the focus is not on new features, but on a big cleanup of the OS infrastructure. Not only this are great news for us, developers, but also shows some courage from Apple and a lot of respect for their users. They want to focus the next year on improving the quality of their OS, rather than packing it up with some new features just to win the race against the competition.
• As a side note, the MacTech people was giving away some magazines for free to the people who were standing in line during the morning. I took the time to read most of it during my flight, and I really liked it. I was a MacTech subscriber in the past, but I cancelled it because, during my graduation, I didn’t have time to read it (it’s good to graduate on a place where you actually don’t have time to learn, isn’t it?). Maybe I’ll subscribe it again now.
• Not related to the conferences themselves, we went to visit the bay area surroundings on Friday afternoon. We did the classic trip to the Apple and Google campuses, because we are all geeks, but we also went to the Stanford and Berkeley campuses. The Standford campus totally blow me away. You have to see it to believe it. From now on, I’ll laugh, really laugh, every time I hear a faculty from my university stating that we actually have a campus. The Berkeley campus did not impress me much. It’s more urban style, more crowded and dense. I prefer the Stanford way, with space, a huge amount of space, tons and tons of space, really. Almost made me want to return to the univ! :)

I installed Parallels and created two virtual machines, one for IE 6 and another one for IE 7. This way I’m sure there are no weird problems between those two versions (having more than an IE version on Windows can only be accomplished by hacks, and hacks are bad). Also I can install Visual Web Developer Express Edition on each of the VMs, and use either IE 6 or 7 to debug.

My apps run inside virtual hosts on Mac OS X apache, under a fake DNS name. On Mac OS X it’s easy to add the DNS entry to the /etc/hosts file, under the 127.0.0.1 entry. This way, your DNS name will always point to your mac, and you’ll be able to reach your virtual host.

I wanted to do the same from inside Windows running on Parallels. An easy way would be to edit the Windows hosts file, adding the Mac OS X public IP to the file. But that will only work if the OS X IP doesn’t change. My Intel mac is an MBP, and I change the network I use often, so I needed a little more flexibility. So, this is the way I found to do this:

1. Configure your VM to use Shared Networking. This wall, Parallels extensions installed on your Mac will create a NAT network where your virtual machine will be hooked into.
   
   
2. Open Mac System Preferences, and look for the “Parallels NAT” network port. This is an interesting one, because it allows the Mac itself to be connected to the virtual NAT network, using an IP on the NAT subnet. Write down that IP: this will be the IP you’ll use to access the Mac virtual hosts from within the virtual machines.
   
   
3. Finally, edit Windows hosts file. This file is located on \WINDOWS\system32\drivers\etc\hosts. Add a line with the IP (in my case, 10.211.55.2) and the name of the virtual host, just like you do on the Mac.
   
   

That’s it. Now you can access your Mac virtual hosts from Windows, whatever the Mac IP is. Ick, what’s a Windows screenshot doing in my blog!?


BufferedImage reducedImage = new BufferedImage(newX, newY, BufferedImage.TYPE_INT_RGB);
Graphics2D g = reducedImage.createGraphics();
g.setRenderingHint(RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BICUBIC);
g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
g.setRenderingHint(RenderingHints.KEY_RENDERING, RenderingHints.VALUE_RENDER_SPEED);

g.drawImage(originalImage, 0, 0, newX, newY, 0, 0, originalImage.getWidth(), originalImage.getHeight(), null);
g.dispose();


Although not blazing fast, this is enough for many applications. I could reduce a 7 Mega-Pixels image to something like 250 pixels wide in about one second, or less, in my PowerBook G4. But this was in Tiger.

In Leopard, as some of you may have noticed (and if you have applications deployed on Leopard Server, be aware) this is incredibly slow. When I say slow, I say five minutes, or even more, with the CPU being used at 100% during that time.

There are two reasons that lead to this. The first (which is not a problem in itself, but it’s a cause of the problem): Apple switched from Quartz to Sun2D graphics engine as the default one for Java applications on Leopard. So, all your image manipulation is being done using the Sun pipeline now. This would not be a problem, except for the second reason: the Apple JVM implementation has a bug that is slowing Sun’s pipeline drawImage method to a crawl. Actually, that was not the real reason. I testes this on FreeBSD (using Diablo JDK) and the speed was similar to Leopard’s. Sun2D is REALLY slow, to the point of being useless. I’m now using ImageMagick.

The only solution for now is forcing the application to use Quartz engine. You can do that using the command line option -Dapple.awt.graphics.UseQuartz=true. And, of course, file a bug on this!

Some presentations were very superficial, but the most interesting ones went as deep as some of the WWDC sessions I attended. The event was actually a micro-WWDC, and even included a nice buffet with plenty of food for lunch and coffe-breaks, all for free. As I spend almost all the WWDC week on the IT track, it was cool to learn about the desktop stuff Apple is working on.

The room was packed, and people were motivated and participative. The Apple guys really liked that, as they say those are the main factors they use to evaluate how successful and event it, specially when going to a new country for the first time. I know I’ll be there next time!

As I said, I bought a second-hand PowerMac G5 to replace my old G4. When I got the new machine, I run Apple Hardware Test (AHT), using the Extended Test. AHT tested my hardware, including the 2.5 GB of RAM, taking more than two hours (and making a hell of a noise, because during tests, the G5 ventilation system works in failsafe mode, which means, full power). Everything seemed to be fine. Until I installed Retrospect. I use Retrospect to make all my backups at home, and despite all it’s quirks, it always worked fine on the G4. Since I installed it on the G5, I got strange errors (the famous “internal consistentcy check”) and even crashes.

After nailing down all the possibilities (trashing preferences and existing backup sets, reinstalling Retrospect, etc) I suspected it could be an hardware problem, because I was told that the “internal consistentcy check” appears when the backup set contents are corrupted. So, I thought, my hard drive is corrupting data. I duplicated one backup set with about 80 GB, and surprise – after duplicating and running an md5 checksum on it (and diff), the files were different! This was NOT supposed to happen, naturally. So I tried the same thing on my boot drive – same problem. Ops… it’s not the drives. So, if it’s not the drives, and supposing (more exactly, praying) that it was not a motherboard issue, it must be the memory.

All my collegues at IST System Adminstration team use memtest on PCs to test the memory. This great distribution of memtest has a really nice touch: you can burn this on a CD, and boot the PC from it. It takes less that 200K of RAM, so all the other memory will be tested. Unfortunately, it’s not possible to boot it in Macs (not even Intel Macs – I tried it!). So, you must get the Mac OS X version of memtest, and boot the OS in Single User mode (using command-S during the boot sequence). The OS will take about 50 MB of RAM, which is, of course, much worse than the 200K used by the PC version, because those 50 MB will simply not be tested. But it’s better than nothing.

The official site for the Mac OS X version of memtest is here, but unfortunately, the author requires you to pay a small ammount for the download. I don’t like the approach very much because I don’t really know what I’m buying. the author says that, after paying, he sends a password for the encrypted DMG you downloaded. But I cannot download without paying, because the link is no-where. So… what happens when a new version comes out? Do I have to pay it again? Well, anyway, someone else is distributing memtest for OS X for free. Yes, it’s legal, because the software is under GNU license. So, if you don’t want to pay, just click here and grap your own free copy. Happy testing!

By the way, some tests take a lot of time. Let all of them run. Don’t assume the fact that all the “quick” tests passed means your memory is OK. Some problems may only be found with the more complex and slower tests – that’s why they are there. So, let it run. And if you have a G5, get the hell out of there, or use ear-plugs. It won’t be a nice office to work during testing, trust me.

memtest will detetct lots of common problems in memories, and will probably identify more than 99% of the defective memory modules arround. But never forget: it’s impossible to be entirely sure that a memory module is OK, simply because it’s not possible, in a reasonable time frame, to test all the possible combinations of data. Also, memory may pass all the tests in a day, and fail the next day. There are many factors that may trigger a hidden problem in memory modules: temperature, electrical flutuations, the data it contains, age, etc. If you suspect you have a bad memory module, and if you have time, run memtest for several days in a row, using the option to do many passes.

Unfortunately, that’s impossible to do with Mac OS X, at least with it’s built-in software. There are some commerical applications to do that (like TechTool Pro), but I get a little pissed off when I have to spend a lot of money buying a software that does a zillion things when all I want is surface scans, and specially when I could do it with the “old” OS and not with the new powerful UNIX-based one.

Well, Linux has the badblocks command that will do just that: test the disk surface for bad blocks. It’s a simple UNIX command, so I thought there must be a port of that to OS X (and, of course, I could try to compile it in OS X as last resource). After some googling, I found out badblocks is part of the ext2fs tools. And, fortunately, Brian Bergstrand has already done the port to OS X, including a nice installer.

The installer installs all the ext2fs stuff, including an extension that will allow you to access ext2fs volumes on OS X. As always, this is a somewhat risky operation. Personally, I avoid as many extensions as I can, because they run too close to the kernel for me to feel confortable. So, if possible, install it on a secondary OS (like an utility/recover system on an exteral hard drive, or so).

The badblocks command will be installed in /usr/local/sbin/badblocks, and it will probably not be on your PATH, so you have to type the entire path when using, or edit your PATH environment variable.

Usage is simple. First, run the “mount” command, so that you know the device names for the drives you want to test. You can obtain something like this:

arroz% mount
/dev/disk0s3 on / (local, journaled)
devfs on /dev (local)
fdesc on /dev (union)
on /.vol
automount -nsl [142] on /Network (automounted)
automount -fstab [168] on /automount/Servers (automounted)
automount -static [168] on /automount/static (automounted)


The internal hard drive is /dev/disk0 (note that /dev/disk0 is the entire drive, /dev/disk0s3 is a single partition). Imagining you want to test the internal hard drive you would type the command (as root):

badblocks -v /dev/disk0

This would start a read-only test on the entire volume. The -v is the typical verbose setting, so you may follow what’s happening. This will take a long time, depending on the hard drive you use. For a 160 GB hard drive, it took between 2 and 3 hours in a G5 Dual 2 Ghz.

I mention this because time is an important factor when testing hard drives! You should run badblocks on a known-to-be-in-good-condition hard drive, so that you can get the feeling of how fast (or slow) badblocks is. Later, if you test a possibly failing hard drive, and badblocks progresses notably slower, it will probably mean that the hard drive is in bad condition (even if it doesn’t have badblocks).

After running the command, you may get two results: your disk has, or hasn’t badblocks! :) You will see many outputs of a successful surface scan, so I leave here an example of a not-so-successful one:

/usr/local/sbin arroz$ sudo ./badblocks -v /dev/disk0
Password:
Checking blocks 0 to 156290904
Checking for bad blocks (read-only test): 120761344/156290904
120762872/156290904
120762874/156290904
done
Pass completed, 3 bad blocks found.


This is the result of a test on a 160 GB hard drive with 3 bad blocks.

After getting something like this, you may try to run badblocks again, in write mode. Note that this will destroy all the information you have on the hard drive! badblocks won’t copy the information to memory, and than back to disk. It simple destroys it. The point of running a write-enabled badblocks check is forcing the hard drive to remap the damaged sectors. Hard drives have a reserved space to use when bad blocks are found. The bad blocks are remapped to that reserved space, until it fills. And this will only happen on a write. So, run badblocks in write mode, and then again in read-only mode. If badblocks finds no bad blocks, your hard drive is fine (for now). If badblocks still finds bad blocks, it means that there are so many damaged blocks on the disk surface that the reserved area is full. Forget it, and throw the disk away. It’s useless.

While this works to backup all the desktop workstations and laptops of GAEL members, I have a problem with our xServe. It runs Mac OS X Server, and Retrospect will not backup machines with the Server version of Mac OS X with the license we have. To do that, we would have to buy a much more expensive license.

No problem. A server, due to it’s nature, doesn’t have the “sudden disappearing” problem of the laptops, so I can use a “classic” UNIX approach – and my choice was rsync and the –link-dest option. You may read about this option in the rsync manpage, but in case you don’t know, what it does is the following: instead of synchronizing a directory in the usual way, it will create a new directory with a new file tree. But, to save space, it won’t copy the non-updated files from the old tree to the new one. Instead, it creates hard links, so that both entries in the file system point to the same data on the hard drive (to the same inode), thus saving space. So, everytime you update your backup, you will create a new tree, but you will only waste the space required by the files that were updated since the last backup, and some more space for the filesystem structures that support the directory tree. You can use a command like this:


// rotate old dirs
rm -rf /Volumes/Storage/test/test.5
mv /Volumes/Storage/test/test.4 /Volumes/Storage/test/test.5
mv /Volumes/Storage/test/test.3 /Volumes/Storage/test/test.4
mv /Volumes/Storage/test/test.2 /Volumes/Storage/test/test.3
mv /Volumes/Storage/test/test.1 /Volumes/Storage/test/test.2
mv /Volumes/Storage/test/test.0 /Volumes/Storage/test/test.1

/usr/bin/rsync --rsync-path=/usr/bin/rsync -az -E -e ssh --exclude=/dev/\* --exclude=/private/tmp/\* --exclude=/Network/\* --exclude=/Volumes/\* --exclude=/private/var/run/\* --exclude=/afs/\* --exclude=/automount/\* --exclude=/.Spotlight-V100/\* --link-dest="/Volumes/Storage/test/test.1" "root@my.machine.com:/Users/arroz/TestDirectory" "/Volumes/Storage/test/test.0/"


Side note: the -E option (capital E) is an option present on Mac OS X rsync version, that forces rsync to copy all the extended Mac file system attributes, including resource forks. It only exists in Mac OS X 10.4 (Tiger) or newer versions. If you are still using 10.3 (Panther) or older, use rsyncx. Do not use rsyncx with Tiger.

Until about a week ago, my backup machine (an old PowerMac G4) had an external SCSI Raid with 640 GB, and an internal RAID 0 (2 * 80 GB drives), besides the boot disk. All the Retrospect backups were being placed on the external RAID, and the server backups were going to the internal RAID 0. Now, I know it’s living on the edge to backup to a RAID 0. But there was really no more space, and it was a temporary situation, because the new drives for the external RAID were already ordered.

When the new drives arrived, I stored all the backups where I could for some days (640 GB was huge when we purchased the RAID, but today is relatively managable), switched the drives and created a new fresh RAID 5. Formatted it in the HFS+ file system, and copied back all the backups, including the server backups and finally trashed the internal RAID 0.

Some path adjustements on my server backup scripts, and we are back in business. But the RAID free space was getting dramatically shorter every day. I used the ‘ls -i’ command to compare the inodes of files that were supposed to be unchanged from the backup of a day to the other in the next day, and as I suspected, rsync was duplicating all the files, instead of hard-linking them.

After Googling a lot, I could not find answers for this. I tried to see if ‘cp -la’ would successfully create hard links, but to my surprise, I found out that the Mac OS X built-in ‘cp’ command would not support the “l” option. Nice. Before installing the GNU ‘cp’ version (and because I’m lazy and I didn’t want to do that) I started thinking about everything I had done since the new drives arrived. The OS was the same, the rsync command was the same, it worked before, so it had to work now. The only reason why it could be not working was because rsync, somehow, thought that all the files were changing, even when they did not.

Suddenly, the solution poped up in my head. Mac OS X has an option, associated to every HFS+ volume, called “Ignore ownership on this volume”. This is turned off by default on the boot drive, but it’s turned on by default on all the external drives you format. There’s a good reason for this: Mac OS X is a consumer product. And average users want to buy an external drive, store data on it, bring it to another Mac, and read their data. They don’t care if their UID is the same on both machines or not.

But this causes serious problems to rsync. Althought the file system will store the owner of the files, it probably won’t report it to the applications who try to read it (or will mask them to the user who’s trying to access them). Somewhere this information is filtered, between the file system and application layers. So, rsync was not getting the real UID of the files. As the files that came from the server had real UIDs, both UIDs wouldn’t match, and rsync would create a new copy because, from it’s point of view, the file had been changed.

The solution was simple – just going to the machine console, and “Get Info” of the external volume. I turned off the “Ignore ownership on this volume” setting, and rsync started operating normally again.