$words[rand()]

This morn­ing I rebooted my test box run­ning VMware ESXi 3.5 to com­plete the upgrade from Update 3 to Update 4. The hyper­vi­sor came back up, but no guests were run­ning and when I popped open the VI Client it indi­cated that there were no data­s­tores con­fig­ured and it could not find any of the vir­tual machines I had in inven­tory. It saw the inter­nal disks and that they were for­mat­ted VMFS, but would not allow me to do any­thing other than for­mat them over again.

Nor­mally this would have sim­ply annoyed me since I would have lost my test VMs, but they don’t take long to build so I’d have just for­mat­ted them and gone on with my day. Unfor­tu­nately within the last week we had tem­porar­ily moved a crit­i­cal application’s VM to this box and we had not prop­erly recon­fig­ured backup. I could restore from the week old backup, but there would be hell to pay.

Since the VMFS par­ti­tions were clearly vis­i­ble I felt I had a chance, but I’m still new to ESX/ESXi so my first step was to flip over to my always run­ning irssi ses­sion (if you use IRC and do not use screened irssi, go Google it now and enjoy) and ask for help in #shsc and #vmware. #shsc always has a few guys who work on large VMware installs idling, and of course #vmware is obvi­ous. While wait­ing for any input from IRC, I went to Google for my next step. I knew ESXi has the capa­bil­ity to be accessed via SSH, but it’s dis­abled by default, so I looked up how to turn it on. A few min­utes later after bring­ing a mon­i­tor over to the machine and reboot­ing it I had SSH access and could go through sys­tem logs from the com­fort of my laptop.

In /var/log/messages I found two entries ref­er­enc­ing my SATA con­troller which looked inter­est­ing:
May 5 14:34:35 vmkernel: 0:00:06:39.406 cpu0:3616)ALERT: LVM: 4482: vmhba000:0:0:3 may be snapshot: disabling access. See resignaturing section in SAN config guide.
May 5 14:34:35 vmkernel: 0:00:06:39.408 cpu0:3616)ALERT: LVM: 4482: vmhba0:0:0:1 may be snapshot: disabling access. See resignaturing section in SAN config guide.

This infor­ma­tion, after a quick trip to Google, led to VMware’s SAN con­fig­u­ra­tion guide which ref­er­ences sim­i­lar issues occur­ring on SANs, so I tried enabling the res­ig­na­tur­ing option and mag­i­cally my data­s­tores reap­peared. After renam­ing them back to their orig­i­nal names and turn­ing the res­ig­na­tur­ing option back off I had all my data and was able to down­load the disk images and VMX files so I was safe in the event of a major problem.

At this point, I could see my VMs but the VI inven­tory was still con­vinced that they were on the “old dri­ves”, so after a bit more time on Google I dis­cov­ered the Import fea­ture within the data­s­tore browser and I was able to bring the VMs back in and get them boot­ing up.

Screen­shot show­ing my data­s­tores and two VMs running

After con­firm­ing that the VMs I really needed were boot­ing and oper­a­tional, I shut every­thing down to move the server back to its spot in my rack. For­tu­nately every­thing came right back up so the pres­sure was now off.

Now my con­cerns shifted. If this hap­pened once, what’s to stop it from hap­pen­ing again? I needed to fig­ure out why it hap­pened. For­tu­nately at nearly the exact moment I started think­ing about this IRC came through for me. “jidar” in #shsc linked to this thread on VMware’s forum with lit­er­ally the exact same symp­toms. A few posts down was a link to this page which again matched my expe­ri­ence exactly and says that U4 updated a num­ber of SATA dri­vers includ­ing the one for the ICH9 con­troller in my Pow­erEdge and changed the way they appear to the hyper­vi­sor, which led to it not rec­og­niz­ing the dri­ves for what they are.

Right now I’m mod­er­ately annoyed at an update that’s not even enough to earn it a minor ver­sion num­ber bump on a piece of soft­ware intended for enter­prise use hav­ing a change with the poten­tial to cause this, but on the other hand I don’t expect any­one who really cares about reli­a­bil­ity to be using SATA local stor­age. Ah well, I learned a bit about nav­i­gat­ing around ESXi’s internals.

Sjur Usken and San­dro Gauci have dis­cov­ered a major flaw in the SIP imple­men­ta­tions on a wide range of IP phones. The short expla­na­tion is that the phones do not ver­ify where a proxy authen­ti­ca­tion request is com­ing from and hap­pily return the SIP authen­ti­ca­tion infor­ma­tion. It is hashed and salted, but the salt is cho­sen by the attacker, so a set of rain­bow tables would make crack­ing it triv­ial. For full details, check out Sjur’s blog post (which spread fairly rapidly around the VoIP world) and his lat­est post show­ing the trace as he attacked a Cisco 7940 I set up for this purpose.

Until the phone ven­dors release fixed firmware (if they do) the only way to defend your­self from this is to not have phones exposed on pub­lic IP addresses. If they have to be for some rea­son (we all know SIP and NAT really don’t get along, and proper SIP aware NAT devices cost a fair bit) set fire­wall rules that pre­vent the phones from speak­ing SIP to any IPs that aren’t part of your VoIP sys­tem. Alter­na­tively, in the event that every sin­gle phone on your sys­tem is sta­t­i­cally addressed, the reverse could be done at the reg­is­trar side. It wouldn’t stop the attack­ers from find­ing the pass­word, but it would pre­vent them from using it in any way.

The impli­ca­tions of an attacker gain­ing the SIP authen­ti­ca­tion infor­ma­tion are of course severe, once they have that they can imi­tate the attacked phone and make calls to any num­ber of regions poten­tially cost­ing thou­sands of dol­lars in the course of a sin­gle night.

It’s been quite some time since I last posted…

Here’s a quick sum­mary of what’s gone on in my life:

• The bas­tard Probe is still in the garage, sta­tus really unchanged since August.
• I got bored after doing an install in Colum­bus and wan­dered in to a used car dealer.  Some­how I drove home in a 2002 BMW 325i…
• Char­lie and Mary both quit at MV (this was over the course of a few months, not at the same time), which threw me in to a sit­u­a­tion I really did not want to be in.  I kinda pan­iced and nearly quit to do con­tract work.  For­tu­nately, when I went to dis­cuss my 2 weeks the boss offered me a sig­nif­i­cant raise, and after sleep­ing on it I decided stick­ing with what I knew and con­tin­u­ing to ride out the bad towards the good (which seems to be get­ting closer) was the bet­ter plan.
• My par­ents moved to Vir­ginia, mak­ing me the pri­mary “sup­port” for my brother when he’s in Toledo at school.

I might go in to detail on some of those later.  Any­ways, what I came to post follows:

I’ve been get­ting in to a few blogs recently, and this one I just dis­cov­ered today.  All I’ve read so far is inter­est­ing and well writ­ten, here’s a few favorites in no par­tic­u­lar order…

http://​www​.vio​len​tacres​.com/​a​r​c​h​i​v​e​s​/​3​1​9​/​j​u​s​t​-​s​a​y​-​n​o​-​t​o​-​b​a​s​t​a​r​d​-​c​h​i​l​d​ren

http://​www​.vio​len​tacres​.com/​a​r​c​h​i​v​e​s​/​5​9​/​t​w​o​-​p​h​r​a​s​e​s​-​t​h​a​t​-​d​e​s​t​r​o​y​e​d​-​a​m​e​r​i​c​a​n​-​c​u​l​t​ure

http://​www​.vio​len​tacres​.com/​a​r​c​h​i​v​e​s​/​4​8​/​f​o​u​r​-​r​o​o​k​i​e​-​m​i​s​t​a​k​e​s​-​p​e​o​p​l​e​-​m​a​k​e​-​t​h​a​t​-​k​e​e​p​-​t​h​e​m​-​p​oor

http://​www​.vio​len​tacres​.com/​a​r​c​h​i​v​e​s​/​2​5​0​/​t​h​e​-​p​e​n​t​e​c​o​s​t​a​l​-​c​h​u​r​c​h​-​a​n​d​-​t​h​e​-​h​o​l​y​-​g​h​o​s​t​-​w​a​n​t​-​y​o​u​-​t​o​-​w​e​a​r​-​p​i​g​-​p​a​n​t​ies

http://​www​.vio​len​tacres​.com/​a​r​c​h​i​v​e​s​/​2​7​9​/​a​-​p​e​d​o​p​h​i​l​e​-​l​u​r​k​i​n​g​-​b​e​h​i​n​d​-​e​v​e​r​y​-​d​a​r​k​-​c​o​r​ner

edited for linkification…stupid word­press, what else does it think i might want when I post a URL?

I just updated this site with new pages telling about the things I love and a bit about myself as well. Links are at the top of the page.

I’m also try­ing out a new tem­plate which I really like.  I think I’ll keep it.