Amanda: recovering a Mac OS X client

This is the third post about Amanda, an open source backup system for UNIX-based computers. The previous two posts were a general introduction to Amanda inner workings, and instructions for configuring a Mac OS X amanda client.

In this post I’ll explain how to recover from a catastrophic failure, like when a hard drive dies. Although much of the steps are identical, this post is focused on how to recover the entire file system and not a small set of files that an user accidentally deleted. To recover something like that, you can simply run amrecover on the server, recover the files you need and transfer them using SFTP or any other protocol to the client machine.

Continue reading “Amanda: recovering a Mac OS X client”

Amanda: installing a Mac OS X client

In my previous article, I presented Amanda, its basic concepts, and how does it compare to Time Machine. Now, I’ll give you an example of how to install and configure a Mac OS X machine to be an Amanda client. The next post will explain how to properly recover after a catastrophic failure.

As with any UNIX tool, Amanda can be compiled, installed and configured in a lot of different ways. How you should do it depends on your needs, so don’t feel pressured to do everything in the same way I did, as it’s not “the right way”, just one way. Also, everything I describe here should work on the Leopard or Snow Leopard versions of either Mac OS X or Mac OS X Server on Intel or PowerPC Macs. I’m not sure about previous versions of the OS, but you may find more information about those in the Mac OS X installation notes page of the Amanda wiki.

Continue reading “Amanda: installing a Mac OS X client”

Amanda on Mac OS X

Given that Retrospect 8 is essentially a piece of crap, I’ve been searching for an alternative I can use when Time Machine is not an option for backing up Macs. The main two points I’m focused on is reliability and speed. I want a backup system I can trust that won’t take the age of the universe to recover a file.

I’ve been using Amanda for a while now to backup all the Macs in our workgroup (10 machines) and so far I’m nothing but happy. Amanda is an open source backup system for UNIX-based operating systems, Mac OS X included (I believe it can also backup Windows clients, but I couldn’t care less).

Continue reading “Amanda on Mac OS X”

Apple 2009 wish list

It’s a brand new year. So here’s my wish list for Apple:

  • Please fix the wireless driver that causes my Mac to crash about 10% of the times I turn Airport off.
  • Please fix the trackpad driver, or whatever is causing the trackpad to behave strongly erratic during about 30 seconds after waking the Mac up.
  • Please fix the damn copy/paste bug that makes the paste command paste the previously copied object and not the most recent one. This is specially irritating when you cut a piece of text, paste and you realize you are pasting something else, and that your supposedly cut piece of text is lost forever, unless you can undo and get it back.
  • Please fix the irritating bug that causes an iChat window to keep being the active one even after I click Safari, making its window go in front of iChat’s. That’s specially annoying when I type apple-W to close the Safari window, and the ichat one goes away.
  • Please provide replacement keyboards for people who has pre-unibody MacBook Pros that, you know, actually sense a keystroke every time the key goes all the way down, without the need to almost punch the key.
  • Please fix whatever is causing my father’s MacBook Pro to keep waking up and going back to sleep when the lid is closed and the charger on, despite I had already turned off every god damn thing that could wake it up, including the lid open event.
  • Speaking about the charger, please provide chargers where the charge light doesn’t go off for some unknown reason. It still works, but it doesn’t inspire a lot of confidence in it ans it’s safety.
  • Please provide granular updates to Mac OS X Server. Please please please pretty please.
  • Please care a little more about the entreprise and IT markets, namely your own web application technology (WebObjects, of course).

Thank you, guys! You must hate me but you’re nice people anyway. Sometimes.

Granular updates, please!

Today was update day for our Mac OS X Server machine. The story is quick: installed Mac OS X Server 10.5.5 update, fixed WebObjects to go back to 5.3.3, rebooted. Surprise: my customized version of PHP was replaced by a new version! Why was it a surprise? Because neither the Mac OS X Server 10.5.5 Update notes, nor the Mac OS X (Client) 10.5.5 Update notes, nor even the security notes about the 10.5.5 updates mention PHP anywhere.

Apple, please, can we have granular updates for Mac OS X Server, like we do on basically any other decent server OS? Can we, system administrators, have the privilege to make decisions on what and when we want to update? Why don’t you finally understand that the needs of the IT people are a little bit different than the average consumer? Because, you know, the feeling I get is that you are completely clueless about the IT needs and the way things work. Really.

I like FreeBSD more and more…

Upgrading Mac OS X Leopard Server

I was expecting that. Fortunately, I was ready for it. So, I’ll describe some details of upgrading Leopard Server from 10.5.2 to 10.5.4 to avoid you the same trouble.

If you read my previous article about Leopard Server, you know that I had to recompile MySQL, PHP and Apache to PowerPC 32 bits, because I could not use the 64 bits version on our G5 server. The story was a bit complicated, but the idea was that I needed PostgreSQL and MySQL support on PHP. As the MySQL default installation on OS X Server doesn’t include the shared libraries, I had to recompile MySQL to be able to recompile PHP. But I could not make MySQL compile for 64 bits, so I compiled everything (including Apache) to 32 bits.

So, I was expecting it: as soon as I would upgrade my server, the installers would replace Apache with a more recent version, and it would not start because PHP and MySQL stuff was compiled for 32 bits. And it happened.

How to fix it? Simple. You have to do two things: first, remove the PowerPC 64 part from the httpd binary. Second, recompile your customized version of PHP, because PHP gets replaced too.

To remove the PowerPC 64 part of the binary, you don’t need to recompile Apache as I did before. There are two very useful commands you can use: file and lipo. If you do the following commands:

cd /usr/sbin
file httpd

The result will be:

httpd: Mach-O universal binary with 4 architectures
httpd (for architecture ppc7400): Mach-O executable ppc
httpd (for architecture ppc64): Mach-O 64-bit executable ppc64
httpd (for architecture i386): Mach-O executable i386
httpd (for architecture x86_64): Mach-O 64-bit executable x86_64

As you can see, Apple ships httpd compiled for their four possible architectures. In my case, I want to remove the ppc64 one to force httpd being executed in 32 bits. That’s where lipo comes in hand:

lipo -remove ppc64 -output httpd32 httpd
mv httpd httpd64
mv httpd32 httpd

That’s it. The ppc64 part of the binary has been removed, and a new binary was created. I then just switch the binaries, and there you go, Apache running on 32 bits mode on PowerPC:

httpd32: Mach-O universal binary with 3 architectures
httpd32 (for architecture ppc7400): Mach-O executable ppc
httpd32 (for architecture i386): Mach-O executable i386
httpd32 (for architecture x86_64): Mach-O 64-bit executable x86_64

Then I just needed to recompile PHP and everything was back on track.

Also, be careful because the 10.5.4 update will upgrade WebObjects to 5.4.2. If you are still using WO 5.3 like me, you’ll have to reinstall 5.3 again, as described in the Mike’s document.

Migrating to Leopard Server

This was it. I spent the easter weekend migrating GAEL‘s Xserve to Leopard Server. It all went well, although some more or less serious issues poped up.

Our server is used mainly for hosting web content and applications (php, perl, and of course, WebObjects). It also handles some Subversion repositories and some other minor stuff.

I did the standard procedure I always do when migrating a machine to a new OS version: clone the hard drive to an external firewire disk, format the internal hard drive, install new OS and migrate data. I usually use migration assistant, but of course, this is a server, it’s a little more complicated than that.

While my memory is still fresh, here’s some notes about it, not necessarily by any specific order.

RAID Formatting

Our Xserve has two 80 GB hard drives configured in software RAID 1 (mirroring). As Apple sometimes does some tweaks and changes on the RAID software and drivers, I decided to destroy the RAID and create a new one. So I did: I booted from the Leopard Server DVD, destroyed the RAID and tried to make a new one. I had some problems with that, though. Disk Utility was not allowing me to create the new RAID. I don’t recall the error exactly, but it had something to do with not being able to mount a volume or RAID slice. I quit Disk Utility and launched it again, and then the RAID creation went fine. I just love to see those two blue leds blinking in sync!

Installation and System Updates

Installation itself went without any issues. Our Xserve has a graphics card, so it was like any regular desktop Mac, click click choose click and wait. The system installed correctly, rebooted, configuration assistants, answered all the questions, network working, etc. Perfect. Then, I went to grab all the system updates. I installed it this weekend, so I had a few updates waiting, namely the 10.5.2 combo update. Then something weird happened – after installing all the available updates, the machine rebooted 3 times in a row instead of just one. I know some updates that came out lately require 2 reboots in a row, but I never had seen 3. When the server finally came to life, I manually rebooted it again 2 or 3 times more, just to see if it was booting OK. Apparently, everything is fine. I checked the logs, and they were inconclusive. So, does anyone know if 3 reboots in a row is normal for all the updates that came out so far for a G5 Xserve running Leopard?


This is a fast one, but… sshd comes with PermitRootLogin defaulting to “yes”. Oh come on, guys!

User Migration

This is one of the most serious issues that I find with Mac OS X Server migration. I had seen this when migrating from Panther Server to Tiger Server, and it’s still a problem. The thing is: you cannot migrate passwords. You can use Workgroup Manager to export all the user information… except passwords. That means all the user passwords will have to be reset on the new server. Of course, I don’t expect the real passwords to be exported – specially because they are hashed, so it’s impossible to recover them. But the hash itself could be exported and imported again.

This presents a very serious issue to system administrators and users. Of course, if you have thousands of users, you should use multiple LDAP servers dedicated to the authentication services, and you can clone them at will, making sure that you never loose information and the service never stops. But when you have about 30 users like we do, that is overkill. Even so, it’s a real pain in the ass to reset all those passwords, because some users are actually not in our office. They are external users, either from the other university campus (although that’s not too bad, I actually live closer to that campus that the one I work in, so I can drive by and take care of that stuff), or, worse, from people in some companies that are working remotely with us.

I believe migrations like this should be transparent to the user, and this little detail make them very very opaque.

64 bits hell

Having a full 64 bits OS running on a 64 bits machine can only be a good thing, right? Well… maybe not.

I’m a little crazy and my organizational skills might be very well defined by the word “chaos”, but I’m not crazy enough to do this in the space of two days without having tested all this stuff first and document the important details. So, before trashing our G5 Xserve, I grabbed an old PowerMac G4, installed Leopard Server and all the stuff that really needs to work. The most experienced of you should be smiling by now. Although it seems that the only important difference between both CPUs for the matters we are discussing is just speed, there’s a really important one: 32 bits VS 64 bits. The G5 is a full 64 bits CPU, and the G4 is 32 bits. Up to Tiger, this is not a problem at all, because most of the OS was also running in 32 bits. This included most services, like DBs and Apache. On Leopard, everything (or close to that) is compiled to four different architectures: PowerPC 32 and 64 bits, and Intel 32 and 64 bits. We’ll come back to this in a minute.

Mac OS X Server is bundled with MySQL, PHP and Apache, but not with PostgreSQL. As I prefer PostgreSQL to MySQL by far, I tend to use PostgreSQL with all the applications I can, including my own WebObjects applications. So, I compiled and installed PostgreSQL on the server. As I also need PHP applications to access PostgreSQL databases, I had to download PHP source code and recompile it with PostgreSQL support (you gotta love a language where you have to recompile the whole damn thing to add support to a DB…). But, to compile PHP with support to MySQL (and PostgreSQL) I need to have the MySQL headers and dynamic libraries. Well, Mac OS X Server is bundled with MySQL binaries, but not the headers or libraries. As there were no binaries available for PowerPC 10.5 on the MySQL page, I also had to grab the source and recompile all this stuff.

This is where problems started. I recompiled MySQL, and put it working after some struggle (I really hate MySQL). Then I recompiled PHP. Installed it, added the LoadModule directive to the apache config file, and restarted apache. Bum. Explosions. Apache would not start. It said that the PHP module was compiled for the wrong architecture. I started to thing, WTF, are you telling me that my Xserve just compiled PHP… for Intel? Why did this work on the test G4 box? Well, what other architecture could it be? :P I started googling for the problem and I got it: apache is compiled for all the four architectures I referred above, and it always runs with the most appropriate one for the machine. In the Xserve case, it uses the PowerPC 64 binaries. The problem is that PHP had been compiled for 32 bits only. Ok, no problem. Go to PHP dir, make clean, poke around with the environment variables, recompile the thing for 64 bits. Bum. More explosions. Guess what, MySQL was NOT compiled for 64 bits! Ok ok, one more level deep in the stack, go to MySQL directory, blablabla, recompile and… BUM! Yet another explosion. Now this one was more complicated. Apparently some of the libraries on the MySQL source code package were not being compiled for 64 bits. So, no 64 bits MySQL means no 64 bits PHP that means no runnable PHP with 64 bits apache that means falling back to Apple’s branded PHP that means… no PostgreSQL.

From what I saw on the Net, convincing MySQL to compile on 64 bits was not a road I wanted to go into. Also, one of the pages I found about the “wrong architecture” problem when starting Apache actually suggested to go in the opposite direction: grab Apache source code and recompile it in 32 bits. Using the mention configure command (./configure –enable-layout=Darwin –enable-mods-shared=all) I compiled the exactly same Apache version that Apple bundles with Leopard Server, and installed it over the Apple branded one. That made it all work, now on 32 bits. Of course, if you follow this trick, please keep in mind that this may break in future system updates. If some Apple system update replaces apache, it will not start unless you recompile it again for 32 bits only, or remove the PHP module.

This 64 bits mess is actually a very nasty problem, and makes me think what I’m actually gaining in all this. And the answer is: zero. My server has one GB of RAM, and will probably never have more than 4. If, for some reason, we actually need to boost the memory so far, it certainly won’t be because of Apache. It gets me thinking about actually how many people will actually need apache to run in 64 bits mode. If it’s more that 1% or 2% of the Xserve users, I’ll be very amazed. And what do I loose? A lot. Not all the open source projects compile easily in 64 bits mode (I know MySQL that comes with Mac OS X Server is compiled for 64 bits, but for some reason the needed fixes for that are not in the public MySQL source code tree), Apache may stop working at all in the next system update, and I had a lot of extra work. Maybe Apple should provide an easy way to switch this kind of stuff between 32 and 64 bits mode at will. Having only one OS version to all the architectures is interesting, but solving the problems that it creates is not.

Wrapping up

Everything is working now, after an entire weekend spent behind many terminal windows. Unfortunately, I have to say that my opinion about Mac OS X Server is not the best one. I have been working lately with FreeBSD. My experience with FreeBSD is way, way less than the experience I have with Mac OS X, so there are probably many downsides in FreeBSD I had not yet to deal with. That being said, I think Mac OS X Server is a very easy to use OS, as long as you keep using the tools Apple provided. As soon as you need different tools, specially the ones that tinker with Apache, you’ll start regretting liking computers in the first place. And surprisingly, you start to find that it’s actually easier to do it in a FreeBSD server. Every software I installed so far in FreeBSD (including WebObjects) was installed in a very easy and straightforward, painless way. Just browse the ports tree, make install clean and there it is. No crazy problems, everything is made to work with everything. And the default configurations are usually safer than Apple’s.

It makes sense: although FreeBSD guys don’t do beautiful GUIs and assistants, they work hard to make sure the system Works. All of it, including all the ports. And most important, not just it works, but it works together. If I had to use a word to define FreeBSD, I would pick “consistency” without hesitation. Even WebObjects, which does not have an “official” port on the FreeBSD port tree actually installs easier in FreeBSD than in OS X (due to the hard work of Quinton Dolan that created a FreeBSD port of WO). And face it: probably all the software you need exist in the port tree. It’s HUGE. And if it doesn’t, you can always install it using the classic UNIX way.

The Apple way is different. Apple picks a very small range of software, compiles and packages it in a very easy to use OS. It’s really easy, way more than FreeBSD in many ways. The problems appear when you conclude that the bundled software is not enough, and you want to install your own. And when that happens, you are completely on your own. You’ll start fighting Apple sometimes weird configurations and file system structure, you may run in binary architecture incompatibilities like I did, and so on. And you’ll probably need to do this, because what comes bundled with OS X Server is probably far from enough to get the job done.

Testing memory

I wrote some days ago about badblocks for testing a hard drive surface. Now, the same for memory.

As I said, I bought a second-hand PowerMac G5 to replace my old G4. When I got the new machine, I run Apple Hardware Test (AHT), using the Extended Test. AHT tested my hardware, including the 2.5 GB of RAM, taking more than two hours (and making a hell of a noise, because during tests, the G5 ventilation system works in failsafe mode, which means, full power). Everything seemed to be fine. Until I installed Retrospect. I use Retrospect to make all my backups at home, and despite all it’s quirks, it always worked fine on the G4. Since I installed it on the G5, I got strange errors (the famous “internal consistentcy check”) and even crashes.

After nailing down all the possibilities (trashing preferences and existing backup sets, reinstalling Retrospect, etc) I suspected it could be an hardware problem, because I was told that the “internal consistentcy check” appears when the backup set contents are corrupted. So, I thought, my hard drive is corrupting data. I duplicated one backup set with about 80 GB, and surprise – after duplicating and running an md5 checksum on it (and diff), the files were different! This was NOT supposed to happen, naturally. So I tried the same thing on my boot drive – same problem. Ops… it’s not the drives. So, if it’s not the drives, and supposing (more exactly, praying) that it was not a motherboard issue, it must be the memory.

All my collegues at IST System Adminstration team use memtest on PCs to test the memory. This great distribution of memtest has a really nice touch: you can burn this on a CD, and boot the PC from it. It takes less that 200K of RAM, so all the other memory will be tested. Unfortunately, it’s not possible to boot it in Macs (not even Intel Macs – I tried it!). So, you must get the Mac OS X version of memtest, and boot the OS in Single User mode (using command-S during the boot sequence). The OS will take about 50 MB of RAM, which is, of course, much worse than the 200K used by the PC version, because those 50 MB will simply not be tested. But it’s better than nothing.

The official site for the Mac OS X version of memtest is here, but unfortunately, the author requires you to pay a small ammount for the download. I don’t like the approach very much because I don’t really know what I’m buying. the author says that, after paying, he sends a password for the encrypted DMG you downloaded. But I cannot download without paying, because the link is no-where. So… what happens when a new version comes out? Do I have to pay it again? Well, anyway, someone else is distributing memtest for OS X for free. Yes, it’s legal, because the software is under GNU license. So, if you don’t want to pay, just click here and grap your own free copy. Happy testing!

By the way, some tests take a lot of time. Let all of them run. Don’t assume the fact that all the “quick” tests passed means your memory is OK. Some problems may only be found with the more complex and slower tests – that’s why they are there. So, let it run. And if you have a G5, get the hell out of there, or use ear-plugs. It won’t be a nice office to work during testing, trust me.

memtest will detetct lots of common problems in memories, and will probably identify more than 99% of the defective memory modules arround. But never forget: it’s impossible to be entirely sure that a memory module is OK, simply because it’s not possible, in a reasonable time frame, to test all the possible combinations of data. Also, memory may pass all the tests in a day, and fail the next day. There are many factors that may trigger a hidden problem in memory modules: temperature, electrical flutuations, the data it contains, age, etc. If you suspect you have a bad memory module, and if you have time, run memtest for several days in a row, using the option to do many passes.