Major VMWare 3.5 bug

During our weekly maintenance window tonight I had to reboot both of my ESX 3.5 hosts, after which I could not start any VMs! Instead of the VMs starting as they should, I was greeted with a typically-useful VMWare error message (can you hear the sarcasm?) saying “A general system error occurred: Internal Error” Gee, that’s helpful. Thanks VMWare!

I googled that error and by some miracle got a hit to this thread in the VMWare forums. It details how the ESX 3.5 Update 2 patch released last week apparently time bombs on August 12! At that time, licensing expires but you don’t realize it until you need to restart or vmotion a machine, after which it refuses to start because it thinks your license has expired!

The “simple” fix is to set the ESX host’s time back to at least August 10, 2008… which will let your VMs start. That’s the good news. The bad news is that even if you’re not syncing the guest time with the host (that little checkbox in VMWare tools is disabled), the guest VM still needs to get an initial time from the host (because there is no CMOS battery to store a hardware clock, of course) and this can wreak havoc within the guest VM. On my Exchange 2007 VM, for example, the information store wouldn’t even start because the time was off and of course it couldn’t authenticate to AD because the time was outside of the Kerberos 5 minute limit.  Thankfully once the guest time was THEN set back to the correct time Exchange came up fine.

As of this writing there is still no patch from VMWare, and the natives are quite upset in the forums. I’m with them. This type of error that can bring down your organization simply because of a licensing error is utterly inexcusable by VMWare. They deserve all of the shame they will get on this one. How this one got by QA is anyone’s guess, but I’m betting it was a RGE (Resume-Generating Event) for someone…