This morning I rebooted my test box running VMware ESXi 3.5 to complete the upgrade from Update 3 to Update 4. The hypervisor came back up, but no guests were running and when I popped open the VI Client it indicated that there were no datastores configured and it could not find any of the virtual machines I had in inventory. It saw the internal disks and that they were formatted VMFS, but would not allow me to do anything other than format them over again.

Normally this would have simply annoyed me since I would have lost my test VMs, but they don’t take long to build so I’d have just formatted them and gone on with my day. Unfortunately within the last week we had temporarily moved a critical application’s VM to this box and we had not properly reconfigured backup. I could restore from the week old backup, but there would be hell to pay.

Since the VMFS partitions were clearly visible I felt I had a chance, but I’m still new to ESX/ESXi so my first step was to flip over to my always running irssi session (if you use IRC and do not use screened irssi, go Google it now and enjoy) and ask for help in #shsc and #vmware. #shsc always has a few guys who work on large VMware installs idling, and of course #vmware is obvious. While waiting for any input from IRC, I went to Google for my next step. I knew ESXi has the capability to be accessed via SSH, but it’s disabled by default, so I looked up how to turn it on. A few minutes later after bringing a monitor over to the machine and rebooting it I had SSH access and could go through system logs from the comfort of my laptop.

In /var/log/messages I found two entries referencing my SATA controller which looked interesting:
May 5 14:34:35 vmkernel: 0:00:06:39.406 cpu0:3616)ALERT: LVM: 4482: vmhba000:0:0:3 may be snapshot: disabling access. See resignaturing section in SAN config guide.
May 5 14:34:35 vmkernel: 0:00:06:39.408 cpu0:3616)ALERT: LVM: 4482: vmhba0:0:0:1 may be snapshot: disabling access. See resignaturing section in SAN config guide.

This information, after a quick trip to Google, led to VMware’s SAN configuration guide which references similar issues occurring on SANs, so I tried enabling the resignaturing option and magically my datastores reappeared. After renaming them back to their original names and turning the resignaturing option back off I had all my data and was able to download the disk images and VMX files so I was safe in the event of a major problem.

At this point, I could see my VMs but the VI inventory was still convinced that they were on the “old drives”, so after a bit more time on Google I discovered the Import feature within the datastore browser and I was able to bring the VMs back in and get them booting up.

Screenshot showing my datastores and two VMs running

After confirming that the VMs I really needed were booting and operational, I shut everything down to move the server back to its spot in my rack. Fortunately everything came right back up so the pressure was now off.

Now my concerns shifted. If this happened once, what’s to stop it from happening again? I needed to figure out why it happened. Fortunately at nearly the exact moment I started thinking about this IRC came through for me. “jidar” in #shsc linked to this thread on VMware’s forum with literally the exact same symptoms. A few posts down was a link to this page which again matched my experience exactly and says that U4 updated a number of SATA drivers including the one for the ICH9 controller in my PowerEdge and changed the way they appear to the hypervisor, which led to it not recognizing the drives for what they are.

Right now I’m moderately annoyed at an update that’s not even enough to earn it a minor version number bump on a piece of software intended for enterprise use having a change with the potential to cause this, but on the other hand I don’t expect anyone who really cares about reliability to be using SATA local storage. Ah well, I learned a bit about navigating around ESXi’s internals.

Sjur Usken and Sandro Gauci have discovered a major flaw in the SIP implementations on a wide range of IP phones. The short explanation is that the phones do not verify where a proxy authentication request is coming from and happily return the SIP authentication information. It is hashed and salted, but the salt is chosen by the attacker, so a set of rainbow tables would make cracking it trivial. For full details, check out Sjur’s blog post (which spread fairly rapidly around the VoIP world) and his latest post showing the trace as he attacked a Cisco 7940 I set up for this purpose.

Until the phone vendors release fixed firmware (if they do) the only way to defend yourself from this is to not have phones exposed on public IP addresses. If they have to be for some reason (we all know SIP and NAT really don’t get along, and proper SIP aware NAT devices cost a fair bit) set firewall rules that prevent the phones from speaking SIP to any IPs that aren’t part of your VoIP system. Alternatively, in the event that every single phone on your system is statically addressed, the reverse could be done at the registrar side. It wouldn’t stop the attackers from finding the password, but it would prevent them from using it in any way.

The implications of an attacker gaining the SIP authentication information are of course severe, once they have that they can imitate the attacked phone and make calls to any number of regions potentially costing thousands of dollars in the course of a single night.

It’s been quite some time since I last posted…

Here’s a quick summary of what’s gone on in my life:

• The bastard Probe is still in the garage, status really unchanged since August.
• I got bored after doing an install in Columbus and wandered in to a used car dealer.  Somehow I drove home in a 2002 BMW 325i…
• Charlie and Mary both quit at MV (this was over the course of a few months, not at the same time), which threw me in to a situation I really did not want to be in.  I kinda paniced and nearly quit to do contract work.  Fortunately, when I went to discuss my 2 weeks the boss offered me a significant raise, and after sleeping on it I decided sticking with what I knew and continuing to ride out the bad towards the good (which seems to be getting closer) was the better plan.
• My parents moved to Virginia, making me the primary “support” for my brother when he’s in Toledo at school.

I might go in to detail on some of those later.  Anyways, what I came to post follows:

I’ve been getting in to a few blogs recently, and this one I just discovered today.  All I’ve read so far is interesting and well written, here’s a few favorites in no particular order…

http://www.violentacres.com/archives/319/just-say-no-to-bastard-children

http://www.violentacres.com/archives/59/two-phrases-that-destroyed-american-culture

http://www.violentacres.com/archives/48/four-rookie-mistakes-people-make-that-keep-them-poor

http://www.violentacres.com/archives/250/the-pentecostal-church-and-the-holy-ghost-want-you-to-wear-pig-panties

http://www.violentacres.com/archives/279/a-pedophile-lurking-behind-every-dark-corner

edited for linkification…stupid wordpress, what else does it think i might want when I post a URL?

I just updated this site with new pages telling about the things I love and a bit about myself as well. Links are at the top of the page.

I’m also trying out a new template which I really like.  I think I’ll keep it.