After much debugging on the network side, thinking that perhaps there was a misconfigured NIC with the wrong VLAN config, the problem was still happening.

So ssh’ing onto the host, I started to trawl through the log files, and came across the below in the /var/log/vmkwarning file:

Feb 17 13:44:19 vminfraboxvmkernel: 18:00:00:11.865 cpu4:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device “naa.6090a028004f243d08ab44c26687e3dd” – issuing command 0×410002074040
Feb 17 13:44:19 vminfrabox vmkernel: 18:00:00:11.865 cpu4:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Retry world failover device “naa.6090a028004f243d08ab44c26687e3dd” – failed to issue command due to Not found (APD), try again…
Feb 17 13:44:19 vminfrabox vmkernel: 18:00:00:11.865 cpu4:4222)WARNING: NMP: nmp_DeviceAttemptFailover: Logical device “naa.6090a028004f243d08ab44c26687e3dd”: awaiting fast path state update…

This was occuring repeatedly every half hour and the entries above filled the logs solidly for about 2 minutes continuously every half an hour.

After doing some digging on the google, I found out that ESX4 has a bug whereby if you have a duff or old connection to an iSCSI LUN – perhaps one that no longer exists – but you never rescanned to remove it – when the host tries to check the paths every 30 minutes, it finds this duff connection and goes through the motions of trying to find failover paths. The bug is that this causes very brief network loss to your guests.

The fix for me was to simply re-scan my adapter, which removed the old mapping to one of our removed LUNS’s and the problem went away.

So follow the below steps to ensure your guests power on automatically on host startup.

Open vSphere Client

In the left hand pane, highlight your choosen host, then click the configuration tab in the main pane.

Click Virtual Machine Startup/Shutdown under the software Section.

Click Properties in the top right hand corner of the main pane.

Then check Allow virtual machines to start and stop automatically with the system, then change the startup order of your guests in the box below as appropriate.

If you want all to startup with the host, then move them all up to automatic startup.

Now when your host restarts, your guests will come up automatically as well.

However, with VMWare things changed a little. Virtual Switches dont provide the same level of functionality as current physical switches (unless you spend lots of money)

As far as I know basic VMWare switches dont have a command to tell you the arp table and the associated hosts / port.

So what can you do if you get for example an IP address conflict where you only know the mac address of the conflicting host?

Well, I found a superb script which interrogates an ESX host and lists the guest name, alongside the MAC address.

Its as follows – for ease create a shell script and paste the below in.

When you run it you will get output like this:

win2k8server2.vmx
 /vmfs/volumes/49a2845c-2b4514f9-09bc-001f295db10d/win2k8server2/win2k8server2.vmx
displayName = “win2k8server2″
ethernet0.generatedAddress = “00:50:56:97:20:dd”

win2k8server1.vmx
 /vmfs/volumes/49ae5e8c-bb3c9818-e2dd-001d295db10d/win2k8server1/win2k8server1.vmx
displayName = “win2k8server1″
ethernet0.generatedAddress = “00:50:56:97:31:98″

Script as follows

——————————-

#!/bin/bash

for COUNT in $(vmware-cmd -l); do
echo `echo $COUNT|awk -F’/’ ‘{print $6}’`
echo ” “`echo $COUNT`
echo ” “`cat $COUNT|grep “displayName”`
#MAC Address may refer to “Address” instead of “generatedAddress”
echo ” “`cat $COUNT|grep “generatedAddress”`
echo
done

————————

It would seem that when the storage failed, VMWare didnt get a chance to cleanly shut the hosts down, although it did shut them down. This resulted in the virtual swap file being left on the filesystem, so that when the storage was available once again there was a orphaned vswp file. This was preventing the hosts from being powered on, because VMWare was seing this, thinking the hosts were on already and throwing an error.

The fix, manually removing the vswp file sorted this out.

I did it by ssh’ing onto the box, navigating to the datastore and the virtual machine directory, and then for safe measure, moving the vswp file to another name – .vswp.bak Then when I tried to power the machine on, it recreated the vswp file and the machine happily powered back on. Once this was up, I fully deleted the vswp.bak file to tidy things up, and we were back in business.

Note, we’ve also found that you might need to delete / move the hlog file too…

Error parsing the server “YOUR SERVERS IP” “clients.xml” file. Login will continue, contact your system administrator.

You click OK and they you get:

The type initializer for ‘VirtualInfrastructure.Utils.HttpWebRequestProxy’ threw an exception.

You click OK and vSphere fails to launch

After upgrading to Windows 7 you may have suddenly read horrible threads that tell you VMWare vSphere client will not work. Well its all lies. Its true that it wont work out the box, but a quick tweak and you’ll be good to go in minutes.

Follow these steps, which will get you up and working again in a flash.

1:) Obtain a copy of .NET framework system.dll from a non Windows 7 box. In my case I took it from a Windows 2003 server. The file is located at %SYSTEMROOT%\Microsoft.NET\Framework\v2.0.50727 which for me was c:\windows\Microsoft.NET\Framework\v2.0.50727

2:)Create a folder (call it Lib+) in your VMWare vSphere installation to put this file. I created the following (on a 32bit installation) C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\Lib+

A quick summary, you should now have a folder called C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\Lib+ with a file called System.dll from a non windows 7 installation of .NET Framework v2.0.50727

3:) Now go into C:\Program Files\VMware\Infrastructure\Virtual Infrastructure Client\Launcher and edit the file VpxClient.exe.config in textpad or notepad etc..

Add in the following lines just before the final line (which reads </configuration>)

<runtime>
  <developmentMode developerInstallation=”true”/>
</runtime>

Save this file

Now create a batch file called whatever you choose – I choose vSphereLauncer.cmd but it doesnt matter, and put the following in it (on the assumption that the paths are correct etc..) – Note it is 2 seperate lines.

SET DEVPATH=%ProgramFiles%\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\Lib+
“%ProgramFiles%\VMware\Infrastructure\Virtual Infrastructure Client\Launcher\VpxClient.exe”

Thats it!

Run your newly created batch file, and you should no longer get that irritating message!

If this post was useful, please click some of the Ad blocks which help me to keep this site running. Thanks!

Once you have the vSphere CLI, change the the directory where the vSphere binarys are installed, i.e /usr/bin

To configure the communities do the following:

vicfg-snmp.pl –server <hostname> –username <username> –password <password> -c <com1>.

Replace <com1> with the community name you wish to set. Each time you specify a community with this

command, the setings you specify overwrite the previous configuration. To specify multiple communities,

separate the community names with a comma.

For example, to set the communities public and internal on the host host.example.com, you might type

vicfg-snmp.pl –server host.example.com –username user –password password -c public, internal.

To configure the SNMP agent traps do:

vicfg-snmp.pl –server <hostname> –username <username> –password <password> -t <target

address>@<port>/<community>.

Replace <target address>, <port>, and <community> with the address of the target system, the port number

to send the traps to, and the community name, respectively. Each time you specify a target with this

command, the settings you specify overwrite all previously specified settings. To specify multiple targets,

separate them with a comma.

For example, to send SNMP traps from the host host.example.com to port 162 on target.example.com using

the public community, type

vicfg-snmp.pl –server host.example.com –username user –password password -t

target.example.com@162/public.

Enable the SNMP agent by typing

vicfg-snmp.pl –server <hostname> –username <username> –password <password> –enable.

Send a test trap to verify that the agent is configured correctly by typing

vicfg-snmp.pl –server <hostname> –username <username> –password <password> –test.

The agent sends a warmStart trap to the configured target.

To configure the SNMP agent for polling do:

vicfg-snmp.pl –server <hostname> –username <username> –password <password> -p <port>.

Replace <port> with the port for the embedded SNMP agent to use for listening for polling requests.

If the SNMP agent is not enabled, enable it by typing

vicfg-snmp.pl –server <hostname> –username <username> –password <password> –enable.

And there you have it SNMP on ESX4i.

First you will need to make the neccesary hole in the ESX firewall, do this by running as root

esxcfg-firewall -e snmpd

Then configure snmp. You will need to do this appropriatly for your environment, but edit /etc/snmp/snmpd.conf

Once you have done this, type

chkconfig –level 2345 snmpd on

To enable snmpd on startup init levels 2, 3, 4 and 5.

Then either reboot or type

/etc/init.d/snmpd start

Jobs a goodun.

Well VMWare has esxtop

Commands when in esxtop are:

s - change the screen update delay – values from 0.1 seconds upwards

If you hit space, it will immediatly update the screen.

f or F allows you to add additional fields / columns

c allows you to view CPU resource utilization

m allows you to view memory resource utilization

d allows you to view storage adapter resource utilization

u allows you to view storage device resource utilization

v allows you to view VM resource utilization

n allows you to view network resource utilization.

All these commands can help you to identify any potential bottlenecks or areas you need to make some additional resource changes.

On the physical console of the ESX 4i hit ALT+F1 to view the console window, i.e you need your keyboard and monitor plugged into your ESX4i server to do this..

Type the word unsupported and hit ENTER.

It will come up with a load of red text, which should begin – You have activated Tech Support Mode.

It will then prompt you for the root password – enter that and you have activated SSH (until you reboot!)

Now to make it permanent.

at the prompt, type vi /etc/inetd.conf

This will edit the script that starts enabled services on bootup.

Using the arrow keys, go down until you find the line that begins with ssh. Note you should see 2 lines, the first begins with ssh, the 2nd begins with #ssh.

Make sure your cursor is right at the start of the line beginning with ssh.

Type i and then press the hash key to comment out the first line (Note you may find your keyboard is not english UK, so you might need to do shift 3 for your hash sign)

Now use the arrows to go down to the next line, the one that originally began #ssh.

Move the cursor so its under the first letter s and press backspace, which should remove the # sign.

Now type :wq and press enter, this will write out the changes you have just made.

Either reboot, or type /sbin/services.sh restart

Job should be done, and you should be able to ssh to your server. Bear in mind the shell you get is not a full linux install, its the vmware console which only has limited features, but very usefull for exploring the filesystem, and moving files around etc..

Have fun.

Trying to convert a VMWare Server machine to a VMware Infrastructure machine the new ESX4i one using VMWare vCentre Converter standalone and I got the above error – “Session not authenticated”

Did a quick bit of research, and found that in task manager I for some reason had VMWare vCentre converter standalone running twice. I didnt start it twice, but the first time I tried the convert it had failed, so I guess for some reason it started another instance.

Killed both processes off restarted VM Converter and all was happy.