This morning I rebooted my test box running VMware ESXi 3.5 to complete the upgrade from Update 3 to Update 4. The hypervisor came back up, but no guests were running and when I popped open the VI Client it indicated that there were no datastores configured and it could not find any of the virtual machines I had in inventory. It saw the internal disks and that they were formatted VMFS, but would not allow me to do anything other than format them over again.

Normally this would have simply annoyed me since I would have lost my test VMs, but they don’t take long to build so I’d have just formatted them and gone on with my day. Unfortunately within the last week we had temporarily moved a critical application’s VM to this box and we had not properly reconfigured backup. I could restore from the week old backup, but there would be hell to pay.

Since the VMFS partitions were clearly visible I felt I had a chance, but I’m still new to ESX/ESXi so my first step was to flip over to my always running irssi session (if you use IRC and do not use screened irssi, go Google it now and enjoy) and ask for help in #shsc and #vmware. #shsc always has a few guys who work on large VMware installs idling, and of course #vmware is obvious. While waiting for any input from IRC, I went to Google for my next step. I knew ESXi has the capability to be accessed via SSH, but it’s disabled by default, so I looked up how to turn it on. A few minutes later after bringing a monitor over to the machine and rebooting it I had SSH access and could go through system logs from the comfort of my laptop.

In /var/log/messages I found two entries referencing my SATA controller which looked interesting:
May 5 14:34:35 vmkernel: 0:00:06:39.406 cpu0:3616)ALERT: LVM: 4482: vmhba000:0:0:3 may be snapshot: disabling access. See resignaturing section in SAN config guide.
May 5 14:34:35 vmkernel: 0:00:06:39.408 cpu0:3616)ALERT: LVM: 4482: vmhba0:0:0:1 may be snapshot: disabling access. See resignaturing section in SAN config guide.

This information, after a quick trip to Google, led to VMware’s SAN configuration guide which references similar issues occurring on SANs, so I tried enabling the resignaturing option and magically my datastores reappeared. After renaming them back to their original names and turning the resignaturing option back off I had all my data and was able to download the disk images and VMX files so I was safe in the event of a major problem.

At this point, I could see my VMs but the VI inventory was still convinced that they were on the “old drives”, so after a bit more time on Google I discovered the Import feature within the datastore browser and I was able to bring the VMs back in and get them booting up.

Screenshot showing my datastores and two VMs running

After confirming that the VMs I really needed were booting and operational, I shut everything down to move the server back to its spot in my rack. Fortunately everything came right back up so the pressure was now off.

Now my concerns shifted. If this happened once, what’s to stop it from happening again? I needed to figure out why it happened. Fortunately at nearly the exact moment I started thinking about this IRC came through for me. “jidar” in #shsc linked to this thread on VMware’s forum with literally the exact same symptoms. A few posts down was a link to this page which again matched my experience exactly and says that U4 updated a number of SATA drivers including the one for the ICH9 controller in my PowerEdge and changed the way they appear to the hypervisor, which led to it not recognizing the drives for what they are.

Right now I’m moderately annoyed at an update that’s not even enough to earn it a minor version number bump on a piece of software intended for enterprise use having a change with the potential to cause this, but on the other hand I don’t expect anyone who really cares about reliability to be using SATA local storage. Ah well, I learned a bit about navigating around ESXi’s internals.

Over the coming weeks I will be spending one week each with a number of PC-based router/firewall products installed as the primary NAT gateway at my apartment. I will be reviewing them based on overall performance, interoperability with my SIP-based VoIP service, QoS capabilities, VPN capabilities, and any extra features that make them stand out from the crowd.

The test platform will be a Dell PowerEdge SC430 with a 1.6 GHz Intel Xeon dual core processor and 4GB of RAM. The current list of software to test is as follows:

• pfSense 1.2 (Stable)
• pfSense 2.0 (Alpha)
• Untangle Open Source 6.1
• Vyatta Community 5

I will also be testing “appliance” type routers based on what is available to me, which currently is as follows:

• Linksys WRT54GL (Linksys firmware 4.30.12)
• Linksys WRT54GL (Tomato 1.23)
• Linksys WRT54GL (DD-WRT v24 SP1 Mega)
• Linksys WRT54GL (OpenWRT Kamikaze 8.09)
• Cisco 1841 (IOS 12.4(23))
• Watchguard Firebox X Edge
• Edgewater Edgemarc 4500 (VOS 9.1.2)

The Watchguard is currently unknown due to not having the password for it and I may cut down the list of Linksys firmwares, but all of the rest will be tested.

Hardware or software suggestions for further testing are appreciated.

Sjur Usken and Sandro Gauci have discovered a major flaw in the SIP implementations on a wide range of IP phones. The short explanation is that the phones do not verify where a proxy authentication request is coming from and happily return the SIP authentication information. It is hashed and salted, but the salt is chosen by the attacker, so a set of rainbow tables would make cracking it trivial. For full details, check out Sjur’s blog post (which spread fairly rapidly around the VoIP world) and his latest post showing the trace as he attacked a Cisco 7940 I set up for this purpose.

Until the phone vendors release fixed firmware (if they do) the only way to defend yourself from this is to not have phones exposed on public IP addresses. If they have to be for some reason (we all know SIP and NAT really don’t get along, and proper SIP aware NAT devices cost a fair bit) set firewall rules that prevent the phones from speaking SIP to any IPs that aren’t part of your VoIP system. Alternatively, in the event that every single phone on your system is statically addressed, the reverse could be done at the registrar side. It wouldn’t stop the attackers from finding the password, but it would prevent them from using it in any way.

The implications of an attacker gaining the SIP authentication information are of course severe, once they have that they can imitate the attacked phone and make calls to any number of regions potentially costing thousands of dollars in the course of a single night.