VMWare ESX4 guests loosing network connectivity briefly.

Came across a very odd issue lately where guests on one of our ESX4 hosts were periodically loosing network connectivity very briefly – maybe 10 ICMP packets every half hour or hour.

After much debugging on the network side, thinking that perhaps there was a misconfigured NIC with the wrong VLAN config, the problem was still happening.

So ssh’ing onto the host, I started to trawl through the log files, and came across the below in the /var/log/vmkwarning file:

Feb 17 13:44:19 vminfraboxvmkernel: 18:00:00:11.865 cpu4:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device “naa.6090a028004f243d08ab44c26687e3dd” – issuing command 0×410002074040
Feb 17 13:44:19 vminfrabox vmkernel: 18:00:00:11.865 cpu4:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device “naa.6090a028004f243d08ab44c26687e3dd” – failed to issue command due to Not found (APD), try again…
Feb 17 13:44:19 vminfrabox vmkernel: 18:00:00:11.865 cpu4:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Logical device “naa.6090a028004f243d08ab44c26687e3dd”: awaiting fast path state update…

This was occuring repeatedly every half hour and the entries above filled the logs solidly for about 2 minutes continuously every half an hour.

After doing some digging on the google, I found out that ESX4 has a bug whereby if you have a duff or old connection to an iSCSI LUN – perhaps one that no longer exists – but you never rescanned to remove it – when the host tries to check the paths every 30 minutes, it finds this duff connection and goes through the motions of trying to find failover paths. The bug is that this causes very brief network loss to your guests.

The fix for me was to simply re-scan my adapter, which removed the old mapping to one of our removed LUNS’s and the problem went away.

VMWare ESX4i guests power on automatically when host restarts.

From ESX3.5i to vSphere VMWare has changed slightly the location of where to configure your host so that the guests restart if you happen to have either power failures to your ESX host or if you just shut it down etc..

So follow the below steps to ensure your guests power on automatically on host startup.

Open vSphere Client

In the left hand pane, highlight your choosen host, then click the configuration tab in the main pane.

Click Virtual Machine Startup/Shutdown under the software Section.

Click Properties in the top right hand corner of the main pane.

Then check Allow virtual machines to start and stop automatically with the system, then change the startup order of your guests in the box below as appropriate.

If you want all to startup with the host, then move them all up to automatic startup.

Now when your host restarts, your guests will come up automatically as well.

Find vmware guest using the MAC Address on ESX

With traditional networking, using switches arp tables it always was pretty simple to find a server using only the mac address.

However, with VMWare things changed a little. Virtual Switches dont provide the same level of functionality as current physical switches (unless you spend lots of money)

As far as I know basic VMWare switches dont have a command to tell you the arp table and the associated hosts / port.

So what can you do if you get for example an IP address conflict where you only know the mac address of the conflicting host?

Well, I found a superb script which interrogates an ESX host and lists the guest name, alongside the MAC address.

Its as follows – for ease create a shell script and paste the below in.

When you run it you will get output like this:

win2k8server2.vmx
 /vmfs/volumes/49a2845c-2b4514f9-09bc-001f295db10d/win2k8server2/win2k8server2.vmx
displayName = “win2k8server2″
ethernet0.generatedAddress = “00:50:56:97:20:dd”

win2k8server1.vmx
 /vmfs/volumes/49ae5e8c-bb3c9818-e2dd-001d295db10d/win2k8server1/win2k8server1.vmx
displayName = “win2k8server1″
ethernet0.generatedAddress = “00:50:56:97:31:98″

Script as follows

——————————-

#!/bin/bash

for COUNT in $(vmware-cmd -l); do
echo `echo $COUNT|awk -F’/’ ‘{print $6}’`
echo ” “`echo $COUNT`
echo ” “`cat $COUNT|grep “displayName”`
#MAC Address may refer to “Address” instead of “generatedAddress”
echo ” “`cat $COUNT|grep “generatedAddress”`
echo
done

————————