Sunsetsurf Techblog

VMWare ESX4 guests loosing network connectivity briefly.

Came across a very odd issue lately where guests on one of our ESX4 hosts were periodically loosing network connectivity very briefly – maybe 10 ICMP packets every half hour or hour.

After much debugging on the network side, thinking that perhaps there was a misconfigured NIC with the wrong VLAN config, the problem was still happening.

So ssh’ing onto the host, I started to trawl through the log files, and came across the below in the /var/log/vmkwarning file:

Feb 17 13:44:19 vminfraboxvmkernel: 18:00:00:11.865 cpu4:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device “naa.6090a028004f243d08ab44c26687e3dd” – issuing command 0×410002074040
Feb 17 13:44:19 vminfrabox vmkernel: 18:00:00:11.865 cpu4:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device “naa.6090a028004f243d08ab44c26687e3dd” – failed to issue command due to Not found (APD), try again…
Feb 17 13:44:19 vminfrabox vmkernel: 18:00:00:11.865 cpu4:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Logical device “naa.6090a028004f243d08ab44c26687e3dd”: awaiting fast path state update…

This was occuring repeatedly every half hour and the entries above filled the logs solidly for about 2 minutes continuously every half an hour.

After doing some digging on the google, I found out that ESX4 has a bug whereby if you have a duff or old connection to an iSCSI LUN – perhaps one that no longer exists – but you never rescanned to remove it – when the host tries to check the paths every 30 minutes, it finds this duff connection and goes through the motions of trying to find failover paths. The bug is that this causes very brief network loss to your guests.

The fix for me was to simply re-scan my adapter, which removed the old mapping to one of our removed LUNS’s and the problem went away.

Howto: extract files from a .msi file

Every once in a while I need to extract the content of a .msi file in order to customize a deployment for a particular network environment. Sometimes initializing the .msi installer will temporarily extract the files into C:\Documents and Settings\<username>\Local Settings\Temp, but those files are removed once the installer exits.

To extract files from a .msi file at the command line, type:

msiexec /a PathToMSIFile /qb TARGETDIR=DirectoryToExtractTo

For example, to extract files from c:\installer.msi into c:\install you would type:

msiexec /a c:\installer.msi /qb TARGETDIR=c:\install

The destination directory does not need to exist prior to running this command.

If this doesnt work, a crude way is to rename the msi file to a rar file, winrar will then allow you to extract all the files – however… it doesnt name them properly, so I used a combination of LesMsierables a .net file which allows you to view msi files and compared the size to find the specific file I wanted..

Hyperterminal / command line modem calls

Not something you use much these days, but something I thought i’d note down here in case someone at some point needs it.

ATA (ata) Handshake sounds if modem speaker is enabled. Modem tries to answer call with handshake sounds. Press Enter to stop this process. The no carrier message is normal with this command. This command does not apply to Windows XP.

ATDT <ISP Phone Number>

For example, ATDT1234567 (where 1234567 is the connection number to dial for Internet service).

To connect to a test server in the US you can type:

ATDT18666633665 Login: _ Connects to a test server. If you can get to a login, your modem is working correctly.

ATDT (atdt) Dialtone if modem speaker is enabled. Indicates that the modem can get a dial tone. The message “no dial tone” indicates that either the modem or phone line is not working correctly. Press Enter to stop this process.

AT&F OK Resets most modems to factory defaults.

ATI0 (ati) 56000 ok The response varies between modems, but returns the modem speed..

ATI3 (ati) <Version and name> Returns the modem product type and firmware version.

ATZ (atz) ok This command resets the modem.

ATH (ath) ok Hangs up modem

ATS0=2 (ats0=2)

Zero, not the letter O . ok This command sets the modem to receive. Call the computer’s phone number and you should hear the handshake signal. Ring, ring displays on HyperTerminal screen, followed by no carrier , when the caller hangs up.