Archive for : July, 2011


Esxtop Guide

Someone once likened esxtop to windows task manager on steroids. When you have a look at the monitoring options available you can see why someone would think that.

Esxtop is based on the ‘nix top system tool, used to monitor running applications, services and system processes. It is an interactive task manager that can be used to monitor the smallest amount of performance metrics of the ESX(i) host.

Esxtop works on ESX and it’s cousin remote esxtop or resxtop is for ESXi, although resxtop will work on ESX as well.

To run esxtop from ESX all you need to do is type esxtop from the command prompt.
To run resxtop you need to setup a vMA appliance and then run resxtop –server and enter the username and password when prompted. You don’t need root access credentials to view resxtop counters, you can use vCenter Server credentials.

The commands for esxtop and resxtop are basically the same so I will describe them as one.

Monitoring with resxtop collects more data than monitoring with esxtop so CPU usage may be higher. This doesn’t mean this esxtop isn’t resource intensive, quite the opposite. In fact it can use quite a lot of CPU cycles when monitoring in a large environment, if possible limit the number of fields displayed. Fields are also known as columns and entities (rows) You can also limit the CPU consumption by locking entities and limiting the display to a single entity using l.

Esxtop uses worlds and groups as the entities to show CPU usage. A world is an ESX Server VMkernel schedulable entity, similar to a process or thread in other operating systems. It can represent a virtual machine, a component of the VMkernel or the service console.  A group contains multiple worlds.

Counter values
esxtop doesn’t create performance metrics, it derives performance metrics from raw counters exported in the VMkernel system info nodes.  (VSI nodes) Esxtop can show new counters on older ESX systems if the raw counters are present in VMkernel.  (i.e. 4.0 can display 4.1 counters)
Many raw counters have static values that don’t change with time. A lot of counters increment monotonically, esxtop reports the delta for these counters for a given refresh interval. For instance the counters for CMDS/sec, packets transmitted/sec and READS/sec display information captured every second.

Counter normalisation
By default counters are displayed for the group, in group view the counters are cumulative whereas in expanded view, counters are normalised per entity. Because of the cumulative stats, percentage displayed can often exceed 100%. To view expanded stats, press e.

esxtop takes snapshots. Snapshots are every 5 seconds by default. To display a metric it takes two snapshots and makes a comparison of the two to display the difference. The lowest snapshot value is 2 seconds. Metrics such as %USED and %RUN show the CPU occupancy delta between successive snapshots.

The manual for (r)esxtop is full of useful examples, and unlike normal commands run on ESXi it will work if you run it from the vMA in the same way that it does from ESX.

To use just type man esxtop or man resxtop.

Commands for (r)esxtop can be split into two distinct types, running commands and interactive commands

Running commands
These are commands that get placed at the end of the initial call of (r)esxtop.
#esxtop -d 05

This command would set a delay on the refresh rate of 5 seconds.

Running commands examples
-d – sets delay.
-o – sets order of Columns.
-b – batch mode – I will explain this in further detail below.

Interactive commands
These are commands that work on a key press once (r)esxtop is up and running.  If in any doubt of which command to use just hit h for help.

Interactive commands examples
c – CPU resource utilisation.
m – memory resource utilisation.
d – storage (disk) adapter resource utilisation.
u – storage device resource utilisation.
v – storage VM resource utilisation.
f – displays a panel for adding or removing statistics columns to or from the current panel.
n – network resource utilisation.
h – help.
o – displays a panel for changing the order of statistics.
i – interrupt resource utilisation.
p – power resource utilisation.
q – quit.
s – delay of updates in seconds. (can use fractional numbers)
w – write current setup to the config file. (esxtop4c)

From the field selection panel, accessible by pressing o you can move a field to the left by pressing the corresponding uppercase letter and you can move a field to the right by pressing the corresponding lowercase letter.  The currently active fields are shown in uppercase letters, and with an asterix to show it is selected in the order field selection screen.

Batch mode (-b)
Batch mode is used to create a CSV file which is compatible with Microsoft perfmon and ESXplot. For reading performance files ESXplot is quicker than perfmon. CSV compatibility requires a fixed number of columns on every row. Because of this reason statistics of vm (world) instances that appear after starting the batch mode are not collected. Only counters that are specified in the configuration file are collected. Using the -a option collects all counters. Counters are named slightly differently to be compatible with perfmon.
To use batch mode select the columns you want in interactive mode and then save with W, then run
#esxtop -b -n 10 > esxtopfilename.csv

-a – all.
-b – Batch mode.
-c – User defined configuration file.
-d – Delay between statistics snapshots. (minimum 2 seconds)
-n – Number of iterations before exiting.

To read the batch mode output file is to load it in Windows perfmon.
(1) Run perfmon
(2) Type “Ctrl + L” to view log data
(3) Add the file to the “Log files” and click OK
(4) Choose the counters to show the performance data.

Esxtop reads it’s default configuration from a .esxtopp4rc file. This file contains 8 lines. The first 7 lines are upper and lowercase letters to specify which fields appear, default is CPU, memory, storage, adapter, storage device, virtual machine storage, network and interrupt. The 8th line contains other options. You can save configuration files to change the default view with in esxtop.

Replay mode
Replay mode interprets data that is collected by issuing the vm-support command and plays back the information as esxtop statistics.  replay mode accepts interactive commands until no more snapshots are collected by vm-support.  Replay mode does not process the output of batch mode.

To use replay mode run
#vm-support -S -i 5 -d 60

untar the file using
#tar -xf /root/esx*.tgz

then run
#esxtop -R root/vm-support*

-S – Snapshot mode, prompts for the delay between updates, in seconds.
-R – Path to the vm-support collected snapshot’s directory.

CPU load average of 1.00 means full utilisation of all CPU’s. A load of 2.00 means the host is using twice as many physical CPU’s as are currently available, likewise 0.50 means half are being utilised.

CPU Screen
– Physical hardware execution context. Can be a physical CPU core if hyperthreading is unavilable or disabled or a logical CPU (LCPU) or SMT thread if hyperthreading is enabled. This displays PCPU percentage of CPU usage when averaged over all PCPUs.
PCPU UTIL(%) – Physical CPU utilised. (real time) Indicates how much time the PCPU was busy, in an unhalted state, in the last snapshot duration. Might differ from PCPU USED(%) due to power management technologies or hyperthreading.

If hyper threading is enabled these figures can be different, likewise if the frequency of the PCPU is changed due to power management these figures can also be adjusted.

As an example if PCPU USED(%) is 100 and PCPU UTIL(%) is 50 this is because hyper threading is splitting the load across the two PCPUs. If you then look in the vSphere client you may notice that CPU usage is 100%. This is because the vSphere client will double the statistics if hyperthreading is enabled.
In a dual core system, each PCPU is charged by the CPU scheduler half of the elapsed time when both PCPUs are busy.

CCPU(%) – Total CPU time as reported by ESX service console. (Not applicable to ESXi)
us – Percentage user time.
sy – Percentage system time.
id – Percentage idle time.
wa – Percentage wait time.
cs/sec – Context switches per second recorded by the ESX Service Console.

CPU panel statistics (c)
ID – resource pool or VM ID of the running worlds resource pool or VM or world ID of running world.
GID – Resource pool ID of the running worlds resource pool or VM.
NAME – err… name.
NWLD – Number of members in a running worlds resource pool or VM.
%USED – CPU core cycles used.
%RUN – CPU scheduled time.
%SYS – Time spent in the ESX(i) VMkernel on behalf of the resource pool, VM or world to processor interrupts.
%WAIT – Time spent in the blocked or busy wait state.
%RDY – Time CPU is ready to run, waiting for something else.

High %RDY and high %USED can imply CPU overcommitment.

Additional fields
%IDLE – As it says. Subtract this from %WAIT to see time waiting for an event. WAIT-IDLE can be used to estimate guest I/O wait time.
%MLMTD (max limited – Time VMkernel didn’t run because it would violate limit settings on the resource pool, VM or world limits setting.
%SWPWT – Wait time for swap memory.

CPU ALLOC – CPU allocation.  Set of CPU statistics made up of the following. (For a world the % are the % of one physical CPU core)
AMIN – Attribute reservation.
AMAX – Attribute limit.
ASHRS – Attribute shares.

SUMMARY STATS – Only applies to worlds.
CPU – Which CPU esxtop was running on.
HTQ – Indicates whether a world is currently quarantined or not. (Y or N)
TIMER/s – Timer rate for this world.
%OVRLP – Time spent on behalf of a different resource pool/VM or world while the local was scheduled. Not included in %SYS.
%CSTP – Time the vCPUS of a VM spent in the co-stopped state, waiting to be co-started. This gives an indication of the co-scheduling overhead incurred by the VM. If this value is high and the CPU Ready time is also high, this represents that the VM has too many vCPUs. If low, then any performance problems should be attributed to other issues and not to the co-scheduling of the VM’s vCPU.

Single key display settings
e – expand. Displays utilisation broken down by individual worlds belonging to a resource pool or VM. All %’s are for individual worlds of a single physical CPU.
u – Sort by %USED column.
r – Sort by %RDY.
n – Sort by GID. (default)
v – VM instances only.
l – Length of NAME column.

CPU clock frequency scaling
%USED – CPU usage with reference to the base core frequency, i.e. the actual CPU value in Mhz.
%UTIL – CPU utilisation with reference to the current clock frequency. (displayed as %)
%RUN – Total CPU scheduled time. Displayed as %. If using turbo boost will show greater than 100%.

%UTIL may be different if turbo boost in enabled. To better define this, if a CPU has a base core clock speed of 1Ghz, with turbo boost is 1.5Ghz then %USED (with turbo boost enabled) is 150%. So the current CPU speed (%UTIL) will be that 150% displayed as a value of 100%. Now if the current used CPU is 1Ghz then the current %UTIL will be 50% and not 100%. (as per the base core frequency) Consider this when monitoring these stats.

Interrupt panel (i)
VECTOR – Interrupt vector ID.
COUNT/s – Interrupts per second on CPU x
TIME/int – Average processing time per interrupt. (in micro seconds)
TIME_x – Average processing time on CPU. (in micro seconds)
DEVICES – Devices that use the interrupt vector. If the interrupt vector is not enabled name is in < > brackets.


The following counters and statistics assume a basic understanding of memory management in a virtualised environment.  Check out my Understanding virtual machine memory guide for a brief overview of memory management.

Memory screen (m)
PMEB(MB) – Machine memory statistics.
Total – Yup you guessed it, total.
COS – Amount allocated to the service console.
VMK – Machine memory being used by the ESX(i) VMkernel.
Other – Everything else.
Free – Machine memory free
VMKMEM(MB) – Statistics for VMkernel in MB.
Managed – Total amount.
Min free – Minimum amount of machine memory VMKernel aims to keep free.
RSVD – Reserved by resource pools.
USVD – Total unreserved.
State – Values are high, soft, hard, low. (Pressure states)
COSMEM(MB) – Statistics as reported by the service console.
Free – Amount of idle memory.
Swap_t – Total swap configured.
Swap_f – Swap free.
r/s is – Rate at which memory is swapped in from disk.
w/s – Rate at which memory is swapped to disk.
NUMA(MB) – Only if running on a NUMA server.
PSHARE (MB) – Page sharing.
shared – Shared memory.
common – Across all worlds.
saving – Saved due to transparent page sharing.
curr – Current.
target – What the ESX(i) system expects the swap usage to be.
r/s – swapped from disk.
w/s – swapped to disk.
MEM CTL(MB) – Balloon statistics.
curr – Amount reclaimed.
target – Host attempt reclaims using the balloon driver, vmmemctl.
max – Maximum amount the host can reclaim using vmmemctl.


AMIN – Memory reservation.
AMAX – Memory limit. A value if -1 means unlimited.
ASHRS – Memory shares.
NHN – Current home node for resource pool or VM. (NUMA only)
NRMEM (MB) – Current amount of remote memory allocated. (NUMA only)
N% L – Current % of memory allocated to the VM or resource pool that’s local.
MEMSZ (MB) – Amount of phyiscal memory allocated to a resource pool or VM.
GRANT (MB) – Guest memory mapped.
SZTGT (MB) – Amount the VMkernel wants to allocate.
TCHD (MB) – Working set estimate.
%ACTV – % guest physical memory referenced by the guest.
%ACTVS – Slow moving version of the above.
%ACTVF – Fast moving.
%ACTVN – Estimation. (This is intended for VMware use only)
MCTL – Memory balloon drive installed or not. (Y/N)
MCTLSZ (MB) – Amount of physical memory reclaimed by ballooning.
MCTLTGT (MB) – Attempts to reclaim by ballooning.
MCTLMAX (MB) – Maximum that can be reclaimed by ballooning.
SWCUR (MB) – Current swap.

m – Sort by group mapped column.
b – sort by group Memch column.
n – sort by group GID column. (Default)
v – Display VM instances only.
l – Display length of the NAME column.


The network stats are arranged per port of a virtual switch.

PORT-ID identifies the port and DNAME shows the virtual switch name. UPLINK indicates whether the port is an uplink. If the port is an uplink, i.e., UPLINK is ‘Y’, USED-BY shows the physical NIC name. If the port is connected by a virtual NIC, i.e., UPLINK is ‘N’, USED-BY shows the port client name.

Network panel (n)
PORT-ID – Port ID.
UPLINK – Uplink enabled.(Y or N)
UP – Guess what.
SPEED – Link in MB.
FDUPLX – Full duplex.
USED-BY – VM device port user.
DTYP – Virtual network device type. (H=hub, S=switch)
DNAME – Device name.

T – Sort by Mb transmitted.
R – Sort by Mb received.
t – Packets transmitted.
r – Packets received.
N – Port-ID. (default)
L – Length of DNAME column.


Storage Panels
– disk adapter.
– disk device. (also includes NFS if ESX(i) host is 4.0 Update 2 or later)
– disk VM.

An I/O request from an application in a virtual machine traverses through multiple levels of queues, each associated with a resource of some sort, whether that is the guest OS, the VMkernel or the physical storage.  Each queue has an associated latency.
Esxtop shows the storage statistics in three different screens; the adapter screen, device screen and vm screen.
By default data is rolled up to the highest level possible for each screen.  On the adapter screen the statistics are aggregated per storage adapter by default, but the can be expanded to display data per storage channel, target, path or world using a LUN.
On the device screen statistics are aggregated per storage device by default and on the VM screen, statistics are aggregated on a per-group basis.  One VM has one corresponding group, so they are equivalent to per-vm statistics. Use interactive command V to show only statistics related to VMs.

Queue Statistics
AQLEN – The storage adapter queue depth.
LQLEN – The LUN queue depth.
WQLEN – The World queue depth.
ACTV – The number of commands in the ESX Server VMKernel that are currently active. QUED The number of commands queued.
LOAD – The ratio of the sum of VMKernel active commands and VMKernel queued commands to the queue depth.
%USD – The percentage of queue depth used by ESX Server VMKernel active commands.

%USD = ACTV / QLEN * 100%

I/O throughput statistics
CMDS/s – Number of commands issued per second.
READS/s – Number  of read commands issued per second.
WRITES/s – Number of write commands issued per second.
MBREAD/s – MB reads per second.
MBWRTN/s – MB written per second.

I/O latencies
I/O latencies are measured per SCSI command so it is not affected by the refresh interval. Reported latencies are average values for all the SCSI commands issued within the refresh interval window. Reported average latencies can be different on different screens, (adapter, LUN, VM) since each screen accounts for different group of I/O’s.

Latency statistics
This group of counters report latency values.  These are under the labels GAVG, KAVG and DAVG.  GAVG is the sum of DAVG and KAVG.

GAVG – round-trip latency that the guest sees for all IO requests sent to the virtual storage device.(should be under 25)
KAVG – latencies due to the ESX Kernel’s command. should be small in comparison to DAVG DAVG latency seen at the device driver level. includes the roundtrip time between the HBA and the storage. (should be 2 or less)
QAVG – average queue latency. QAVG is part of KAVG (should be zero)

Storage adapter
CID – Channel ID.
TID – Target ID.

e – Expand/rollup storage adapter statistics.
p – Same as e but doesn’t roll up to adapter statistics.
a – Expand/rollup storage channel statistics.
t – Expand/rollup storage target statistics.
r – Sort by READ/s.
w – Sort by WRITES/s.
R – sort by MBREADS/s.
T – Sort by MBWRTN/s.


Further information can be found at Duncan Eppings blog and in the excellent VMware communities post on Interpreting esxtop Statistics.

Distributed Virtual Switch Guide

The vNetwork distributed switch is a vCenter Server managed entity.  It is designed to create a consistent switch configuration across every host in the datacentre. It allows port groups and individual port settings and statistics to migrate between ESX(i) hosts in a datacentre, the uses of private VLAN’s (PVLAN) and third party deployment.
vCenter Server stores the state of distributed ports (dvports) in the vCenter Server database.  What this means is that the networking statistics, policies and settings migrate with the virtual machine when it is moved from host to host.

The distributed virtual switch is a feature available to enterprise plus license holders with vSphere 4.0 or higher.

The vNetwork distributed switch is abbreviated as vDS and is more commonly know as the dvSwitch.
A vSwitch has port groups and ports and the dvSwitch has dvport groups and dvports. This provides a logical grouping which simplifies configuration.  Dvport Groups define how a connection is made through the dvswitch to the network.  Dvports can exist without a dvport group.

The dvswitch consists of two main components, the control plane and the I/O or data plane.

The control plane
This exists in vCenter Server and is responsible for configuring the dvSwitch, dvport groups, dvports, uplinks, VLAN’s, PVLAN’s, NIC teaming and port migration.
As the control plane exists only in vCenter Server and not on local hosts it is important to remember that a dvswitch is not a big switch spanning multiple hosts but rather a configuration template used to set the settings on each host connected to the dvSwitch.  As such two virtual machines connected to two different hosts would not be able to communicate with each other unless the hosts have uplinks that are both connected to the same broadcast domain.
The configuration settings defined on the vCenter Server are on each host are stored in the /etc/vmware/ESX.conf file.

The I/O plane (data plane)
This is the physical input/output switch that exists on each host, it is a hidden switch and is responsible for forwarding the data packets to the relevant uplinks with the forwarding engine and teaming ports on the host with the teaming engine.
Between the virtual NIC (vNIC) in the virtual machine (VM) and the distributed port in the dvSwitch there can also be software filters. These filters are there for things such as vShield Zones and the VMsafe API.
Because of the separation between the control plane and the I/O plane (data plane) the actual data I/O can quite happily run without access to the vCenter server, as such if vCenter goes down the network service will remain uninterrupted. It just means that you cannot add, remove or change any settings on the dvSwitch.

You can use the net-dvs command to display local information about dvSwitches by running:
# /usr/sbin/vmware/bin/net-dvs

Uplink groups
Uplink groups are created to allow for physical hardware abstraction from each host.  This means that each host will contribute physical NICs into the uplink group and then vCenter Server will only see and use these groups of uplinks when configuration changes are made to the dvSwitch.  This is similar to the way that NIC teaming works.  Network policies and rules get applied to the group and each host then translates that information to it’s individual vmnicx.

Dvport groups
Distributed Virtual Port Groups are similar to port groups in that they create port templates that define port policies for similarly configured ports for attachment to VMs; VMkernel ports and Service Console ports.

Port Groups and Distributed Virtual Port Groups define:
VLAN membership
Port security policies (promiscuous mode, MAC address changes, Forged Transmits)
Traffic shaping policies (egress from VM)
NIC teaming policies  (load balancing, failover detection and failback)

In addition to these features and functions, a Distributed Virtual Port Group also defines:
Ingress traffic shaping policies (enabling bi-directional traffic shaping)
Port Blocking policy

Port bindings
The Port bindings on a dvSwitch determine when and how the VM NIC is assigned a port in a port group.

There are three different port binding types
Static – dvport is assigned permanently to a vNIC when added to port group. (Default)
Dynamic – dvport is assigned only when a VM is first powered on after joining the port group.  This allows for overcommitment of dvports.
Ephemeral – no binding.  This is similar to the a standard switch.  Port is assigned to every VM regardless of state.

Private vlans
Private vlans are vlans that are created within other vlans. These are especially useful when you have devices that should only communicate outside of the vlan they are in and not with each other.
They can also be used when you have devices on a shared wireless network where you don’t want various wireless devices to be able to communicate with each other or should you run out of physical ports on a switch and you want to create additional separate vlans without having to purchase additional network cards and switches.

Private vlans operate a three different modes
Promiscuous – devices can communicate with all interfaces on the vlan and the ones the uplink is a member of.
Isolated – can only communicate with the promiscuous port they are connected to, so devices in the private vlan cannot communicate with each other.
Community – devices can communicate with each other and the promiscuous port as well.

There can only be one promiscuous pvlan, this is created automatically when you setup a private pvlan.
In order to create a pvlan you need to follow a few simple steps
Step 1. Create the relevant vlans and private vlans on the physical switch and ensure the uplinks from the dvSwitch are members of both vlans in trunk mode.
Step 2. Modify the dvSwitch so that the relevant vlans and pvlans are set
Step 3. Create a dvport group that uses the pvlans. When doing this choose the relevant vlan type as private vlan to then be given the option of selecting the pvlan id created earlier in step 2.

Eric Sloofs video on configuring private vlans with Cisco switches

CDP or the Cisco discovery protocol works at layer 2 or the OSI model and contains information about neighbouring devices on the network such as routers/switches models, IOS version and port quantities.
Cisco routers and switches can exchange CDP data with vSphere standard and distributed switches.  CDP is disabled by default on vSwitches and enabled by default on dvSwitches.
The information it collects and is updated every one minute.

CDP on a vdSwitch can be set in one of four ways
Down – no send or receive
Listen – receive
Advertise – send
Both – listen and advertise

To enable CDP on a switch
#esxcfg-vswitch -B vswitch0

View CDP on a vSwitch with
#esxcfg-vswitch -b vswitch0

To enable on a dvSwitch, go to advanced settings on the dvSwitch in the networking section of the vCenter Server and tick the box labelled enable CDP.


With the advent of vSphere 4.1 new features were introduced including load-based teaming and network I/O control.

Load-based teaming
This was introduced to add a true network balance on the network adapters in a team. Up until 4.1, teaming in a virtual environment really meant splitting the load across multiple NIC’s, but fundamentally it was not possible to say put two 1GB NIC’s together and expect the load to be split evenly out over the NIC’s, so you could potentially have all traffic travelling over NIC 1 and nothing over NIC 2. NIC 2 would still step in to take out the load if NIC 1 went down, (if configured so) but there would be no guaranteed spilt over the two.
This is what is now available with load-based teaming.

Network I/O control
Network I/O control is used to divvy up the bandwidth I/O on a network adapter, such as a 10GBe NIC based on the priority of the traffic using network shares. As an example you can set vMotion traffic to have larger shares so that it has a fair priority of network time.
This is only available with the dvSwitch.

Creating a distributed switch

  1. In vCenter, click Home > Inventory > Networking.
    Note: If you are in other locations, the New vDS option is disabled
  2. Right click the Datacentre and select New vNetwork Distributed Switch and specify the:
    • Name of the Distributed Switch
    • Number of Uplink Ports. Uplinks can be renamed/added afterwards.
  3. Click Next.
  4. In the Create vNetwork Distributed Switch GUI, under Add hosts and physical adapters:
    1. Click Add now or Add later
    2. Choose the ESX(i) hosts
    3. Select physical adapter to select adapter per ESX(i)
      Note: Click on view details, to see general adapter information and CDP Device and port ID
  5. Click Next and finish.Note: This automatically creates the default dvPort group, which can be deselected.  At this point the newly created dvSwitch is displayed under the DataCentre on the left panel.

Modifing vNetwork Distributed Virtual Switch

    1. Go to Host > Configuration > Networking.
    2. Right-click on the desired dvSwitch and click Edit Settings.
    3. Click General and specify the following:
      • dvSwitch name (VMware recommends not changing this after full deployment)
      • Number of dvUplink ports
      • Important administrator notes
    4. Click Advanced and specify the following:
      Max value for Maximum Transmission Unit (MTU)
      Useful for enabling Jumbo Frames. Supported value is up to 9000.
    5. Cisco Discovery Protocol (CDP) Operation.
      Select from the drop down menu the following options:
Hany Michaels over at has an excellent Visio diagram detailing the functional workings of the dvSwitch.  This is quite possibly the best dvSwitch Visio diagram around.

Working with ESX(i) Log Files

Working with ESX(i) log files is important when troubleshooting issues within the virtual environment. You can view and search log files in ESX(i) and in vCenter Server using a few different methods.

Using the vSphere client
The direct console user interface (DCUI)
A web browser
A syslog or vMA appliance
An SSH connection to the host
PowerCLI using the Get-Log command

When using SSH, use the following commands to view and search the log files.
Use more to page through the log files one page at a time
Use tail to view the end of the log files
Use grep to search
Use pipe | to link commands together
Use pipe | to grep to search through files
Use cat to concatenate & use grep to search
Use find | print | grep filename to search for a file

Cat hostd.log | grep search variable | more

vCenter log files
vCenter log files are in a vpxd-xx.log format where xx is a numerical value that increases when each log file is 5MB in size.
The log file numbers rotate when the vpxd service is started or when the log reaches 5MB in size.
The log files are located in the c:programdataVMwareVMware virtual centerlogs

Other log files include

Esx logs
/var/log/VMkernel – VMkernel messages
/var/log/messages – service console
/var/log/vmware/vpx/vpxa.log – vSphere client agent
/var/log/aam/VMware__xxx.log- HA
/var/log/vmkiscsid.log – iSCSI
/var/log/boot-logs/sysboot.log – boot log

ESXi logs
/var/log/messages – combination of the VMkernel and vmkwarning
/var/log/vmware/hostd.log – host management service
/var/log/vmware/vpx/vpxa.log – vSphere client agent
/var/log/sysboot.log – boot log
/var/log/vmware/aam/vmware__xxx.log – HA

Log rotation
Within ESX(i), rotation for most log files is controlled by /etc/logrotate.conf
To view available options run man logrotate

With both ESX and ESX(i) hostd.log rotation is controlled with /etc/vmware/hostd/config.xml
Vpxa.log rotation is controlled with /etc/opt/VMware/vpxa/vpxa.cfg

Should you wish to edit the rotation control files you can use Nano in ESX or vi in ESX and ESXi.
I will focus on vi as I am more familiar with it and with the arrival of vSphere 5 there will no longer be ESX, and as such no native support for Nano.

vi commands
a – append
i – insert
O/o – open new line – O is line above, o is line below
r – replace
: – search or save options
/ – search
wq – write and quit
x – delete individual characters
dd – delete line
$ – go to the end of the line
ESC – break out of current mode

Log bundles
Log bundles can be accessed through the VMware folder on the start menu, by clicking generate vCenter server log bundle. This runs the vc-support windows scripting file located at c:program filesVMwarevirtual infrastructurevirtual centrescriptsvc-support.wsf and cscript.

You can also download it through the vSphere client and by connecting to the ESX(I) server using scp with Veeam FastSCP or WinSCP. To do this you have to enable tech support mode first.
An alternative way of generating log bundles is through the vm-support command run through an SSH connection to the COS or through the vMA.  Running vm-support will generate a tar compressed file.


With ESXi it is possible to place log files on shared storage.  To set this open the vSphere client connection to the host, click configuration>advanced settings>syslog select local and enter the path to the shared storage.  Enter the log file location as [datastorename]/logfiles/hostname.log.

vilogd is a service that performs log collections.
You can manage it with the vilogger commands.  vilogger is used to enable and disable or configure the log collections with these commands.

To use vilogger, first ensure that vi-fastpass is enabled using vifp list server to list out the current vi-fastpass enabled servers, if no servers are listed use vifp addserver servername and vifptarget -s servername to add again.

vilogger enable
vilogger list
vilogger update policy

Control the vilogd service with etc/init.d/vmware-vilogd start|stop|restart

vilogger has several parameters available, an example of which are
–numrotation number of files to collect
–maxfilesize specified in MB
–collectionperiod how often to poll, specified in seconds

vilogger enable –server servername –numrotation 20 –maxfilesize 10 –collectionperiod 10

This command will collect the following logs from the ESXi host

To scroll through the log files one page at a time use the more command.

more hostd.log

Configure vMA as a Syslog Server
You can configure the vMA as a syslog receiver to collect log files from the ESX and ESXi server.  Run the commands listed below to configure.

#sudo service rsyslog stop
#sudo nano /etc/sysconfig/rsyslog

This will open nano so you can edit the following information
change SYSLOGD_OPTIONS=”-m 0″ to SYSLOGD_OPTIONS=”-r -m 0″

Save and exit the file
#sudo service rsyslog start
#sudo iptables -I INPUT -i eth0 -p udp –dport 514 -j ACCEPT
#sudo nano /etc/rc.local

Edit the file to add the iptables line below to the end of the rc.local file
iptables -I INPUT -i eth0 -p udp –dport 514 -j ACCEPT

To configure ESX to use vMA as a syslog server add the IP address of the vMA to the /etc/syslog.conf  file.
#vi /etc/syslog.conf

Add the following lines to the bottom of the file
# Send all syslog traffic to vMA
*.* @<IP_Address_Of_vMA>

Open the firewall with
#/usr/sbin/esxcfg-firewall -o 514,udp,out,syslog

Finally restart the syslog service with
#sbin/services syslog restart

Use the vSphere client by going to configuration>advanced>syslog settings and enter into the syslog.Remote.Hostname section the name of the vMA.

Alternatively assuming vi-faspass is enabled run
#vifptarget -s [ESXihost]
#vicfg-syslog -s [vMA]
#vifptarget -c

Connecting to an iSCSI SAN with Jumbo Frames enabled

The best way to add iSCSI storage is by utilizing dedicating NIC’s to iSCSI traffic, on dedicated VMkernel switches, with separate IP subnet address ranges and separate physical switches or VLAN’s.

Enable Jumbo Frames on a vSwitch
To enable Jumbo Frames on a vSwitch, change the MTU configuration for that vSwitch.  It is best to start with a new switch when setting this up as you will need to delete the existing port groups in order to allow jumbo frames to pass through the port group.
In order to run the necessary commands connect to the host using the vSphere CLI which can be downloaded from the VMware website.

To run a vSphere CLI command on Windows
Open a command prompt.
Navigate to the directory in which the vSphere CLI is installed.
cd C:Program FilesVMwareVMware vSphere CLIbin3
Run the command, passing in the connection options and any other options.
<command>.pl <conn_options> <params>
The extension .pl is required for most commands, but not for esxcli.

Example –server my_vcserver –username username –password mypwd –vihost my_esxhost –list

Create a new vSwitch and assign the appropriate uplink.
Open the vSphere CLI and run
vicfg-vswitch –server my_vcserver –username username –password mypwd –vihost my_esxhost -m MTU vSwitch command.

This command sets the MTU for all physical NICs on that vSwitch. The MTU size should be set to the largest MTU size among all NICs connected to the vSwitch.
Run the vicfg-vswitch -l command to display a list of vSwitches on the host, and check that the configuration of the vSwitch is correct.

Create a Jumbo Frames-Enabled VMkernel Interface
Use the vSphere CLI to create a VMkernel network interface that is enabled with Jumbo Frames.
On the vSphere CLI, run the vicfg-vmknic command to create a VMkernel connection with Jumbo Frame support.

vicfg-vmknic -a -I ‘ip address’ -n netmask -m MTU ‘port group name’

Check that the VMkernel interface is connected to a vSwitch with Jumbo Frames enabled.
Run the vicfg-vmknic -l command to display a list of VMkernel interfaces and check that the configuration of the Jumbo Frame-enabled interface is correct.
Configure all physical switches and any physical or virtual machines to which this VMkernel interface connects to support Jumbo Frames.

Create Additional iSCSI Ports for Multiple NICs
Log in to the vSphere Client and select the host from the inventory panel.
Click the Configuration tab and click Networking.
Select the vSwitch that you use for iSCSI and click Properties.
Connect additional network adapters to the vSwitch.
In the vSwitch Properties dialog box, click the Network Adapters tab and click Add.
Select one or more NICs from the list and click Next.
with dependent hardware iSCSI adapters, make sure to select only those NICs that have a corresponding iSCSI component.
Review the information on the Adapter Summary page, and click Finish.
The list of network adapters reappears, showing the network adapters that the vSwitch now claims.

Create iSCSI ports for all NICs that you connected.
The number of iSCSI ports must correspond to the number of NICs on the vSwitch.

In the vSwitch Properties dialog box, click the Ports tab and click Add.
Select VMkernel and click Next.
Under Port Group Properties, enter a network label, for example iSCSI, and click Next.
Specify the IP settings and click Next.
When you enter subnet mask, make sure that the NIC is set to the subnet of the storage system it connects to.
Review the information and click Finish.

CAUTION If the NIC you use with your iSCSI adapter, either software or dependent hardware, is not in the same subnet as your iSCSI target, your host is not able to establish sessions from this network adapter to the target.

Map each iSCSI port to just one active NIC.
By default, for each iSCSI port on the vSwitch, all network adapters appear as active. You must override this setup, so that each port maps to only one corresponding active NIC. For example, iSCSI port vmk1 maps to vmnic1, port vmk2 maps to vmnic2, and so on.

On the Ports tab, select an iSCSI port and click Edit.
Click the NIC Teaming tab and select Override vSwitch failover order.
Designate only one adapter as active and move all remaining adapters to the Unused Adapters category.
Repeat the last step for each iSCSI port on the vSwitch.

Configure iSCSI binding to iSCSI adapters
Identify the name of the iSCSI port assigned to the physical NIC. The vSphere Client displays the port’s name below the network label.

In the following graphic, the ports’ names are vmk1 and vmk2.

Use the vSphere CLI command to bind the iSCSI port to the iSCSI adapter.
esxcli swiscsi nic add -n port_name -d vmhba

IMPORTANT For software iSCSI, repeat this command for each iSCSI port connecting all ports with the software iSCSI adapter. With dependent hardware iSCSI, make sure to bind each port to an appropriate corresponding adapter.
Verify that the port was added to the iSCSI adapter.
esxcli swiscsi nic list -d vmhba
Use the vSphere Client to rescan the iSCSI adapter.

This example shows how to connect the iSCSI ports vmk1 and vmk2 to the software iSCSI adapter vmhba33.
1 Connect vmk1 to vmhba33: esxcli swiscsi nic add -n vmk1 -d vmhba33.
2 Connect vmk2 to vmhba33: esxcli swiscsi nic add -n vmk2 -d vmhba33.
3 Verify vmhba33 configuration: esxcli swiscsi nic list -d vmhba33.
Both vmk1 and vmk2 should be listed.

If you display the Paths view for the vmhba33 adapter through the vSphere Client, you see that the adapter uses two paths to access the same target. The runtime names of the paths are vmhba33:C1:T1:L0 and vmhba33:C2:T1:L0. C1 and C2 in this example indicate the two network adapters that are used for multipathing.

The next thing is to configure the switch with the relevant settings. For this I have used two Dell Powerconnect 5448 switches and a Dell EqualLogic PS4000XV SAN, however the information is relevant for most Dell switch and SAN combination and most other brands too. The commands may differ slightly but the principals are the same.

Configuring the iSCSI SAN switches

Turn on flow control on the switches:
console> enable
console (config)#
interface range ethernet all
console (config-if)#
flowcontrol on

Enable Spanning tree and portfast globally
Console (config)# spanning-tree mode rstp
Console (config)#interface range ethernet all
console (config-if)# spanning-tree portfast

Confirm Unicast Storm Control is disabled with
console# show ports storm-control
Should return State Disabled as show in the image

Check iSCSI awareness is enabled using
Console# config
console(config)#iscsi enable

Disable STP on ports that connect SAN end nodes
console (config)#
interface range ethernet g1,g3
console (config)# spanning-tree disable
console (config-if)# exit

Enable LAG between switches
Disconnect switches from each other before doing following config on both.  Then connect ports 5,6,7, and 8
console (config)# interface range ethernet g5,g6,g7,g8
channel-group 1 mode on
interface port-channel 1
flowcontrol on
console(config-if)# exit

Enable jumbo frames on iSCSI ports (This command will enable it on all ports)
console (config)# port jumbo-frame
This setting will take effect only after copying running configuration to startup configuration and resetting the device.

configure VLANS for vMotion
console(config)# vlan database
console(config-vlan)# vlan 2
console(config-vlan)# exit
console(config)# interface vlan 2
console(config-if)# name vMotion
console(config)# interface range ethernet g2,g4
console(config-if)# switchport mode general
console(config-if)# switchport general pvid 2
console(config-if)# switchport general allowed vlan add 2 tagged
console(config-if)# switchport general acceptable-frame-type tagged-only
console(config-if)# exit
console(config)# interface vlan 2
console(config-if)# ip address
console(config-if)# exit
console(config)# exit
console# copy running-config startup-config
Overwrite file [startup-config] ?[Yes/press any key for no].
Console# reload

Log into switch and set name and time synchronisation options.

VMware vMA

The vMA, or vSphere management assistant is an ovf template which can be downloaded for free from the VMware website.
The vMA has the following minimum system requirements
ESX(i) 3.5 update 2 or later
64 bit  Intel-VT or AMD-V compatible CPU as the vMA is a 64-bit virtual machine
5GB virtual disk size
Access to the ESX(i) host management network
An ESX(i) server or a vCenter Server can be used as a target for the vMA.

Once the vMA has been downloaded and imported into ESX(i), power it on and it will run through a setup wizard where you can set the IP address, password  and SNMP settings.
When you get to the prompt, login to the vMA using the vi-admin account.
The vMA contains the vCLI command utilities which offers the vicfg-xxxxxx commands among others.  These commands are located in the /usr/bin directory.
Use command –help to list all available commands.

An example command is
vicfg-nics –list –servertargetservername
Once prompted enter the vi-admin user name and password.
This will list the available nics on the target server.

Other commands available from the vCLI include

When a command is run through the vMA it is necessary to provide credentials each time for the target server.  To get around this you can use to vi-fastpass command to add credentials to the vMA so that future commands will be run using these cached credentials.
To do this to two things are required.
Vifp – to add the server and credentials
Vifptarget – to specify the server that the command is run on.

Vifptarget commands
–set | -s
–clear | -c
–display | -d
–help | -h

To use vifp type
vifp addserver ‘servername’
Type the vi-admin user name and password when prompted
Once added use Vifptarget within the command to tell the vMA the specific host to run commands on.

So to use the earlier example of vicfg-nics type
vifptarget -s servername
vicfg-nics -l
Notice that there is no prompt for credentials this time.
Once the server you are running the commands against has been added with vicfgtarget -s you will be able to run all subsequent commands within this session without specifying the server name of credentials.

Session Files
An alternative to vi-fastpass for automating cached credentials are session files.
Session files are Perl script files that save session credentials in an aptly named saved session file, called save_

Example –serverservername‘  –user name root –password password –savesessionfile /home/vi-admin/esxsession
Once this has been saved, add it to the end of a command run through the vMA.

vicfg-nics -l –sessionfile /home/vi-admin/esxsession

Consolidating log files
ESXi doesn’t save log files.  When the host is shutdown, all log files are deleted so in order to be able to refer to log files after a shutdown or reboot it is necessary to consolidate the log files.  This can be achieved through the vMA using vilogger.
To use vilogger, first ensure that vi-fastpass is enabled using vifp list server to list out the current vi-fastpass enabled servers, if no servers are listed use vicfg addserver servername and vifptarget -s servername to add again.
vilogger has several parameters available, an example of which are
–numrotation number of files to collect
–maxfilesize specified in MB
–collectionperiod how often to poll, specified in seconds

vilogger enable –server servername –numrotation 20 –maxfilesize 10 –collectionperiod 10
This command will collect the following logs from the ESXi host
To scroll through the log files one page at a time use the more command.

more hostd.log

Resetting the password of the vi-admin account.
To reset the vi-admin password, first reboot the vMA using the VMware tools and when it is starting up press a key on the grub loader screen to pause the boot, then press e to edit and add the word single the kernel command line, then continue the boot process.
Once it boots you will be at the sh-3.2# prompt, this means that the server is in single user mode.
Type the following at the prompt
sh-3.2# passwd vi-admin
Enter the new password when prompted and again to confirm.  You have now reset the password and will be able to login when you reboot the vMA.

Additional vicfg Commands
vicfg-nics | Manage physical NICs
vicfg-vswitch | Manage virtual switches
vicfg-vmknic | VMkernel NIC
vicfg-dns | Specify DNS configuration
vicfg-route | Manage default IP gateway
Vicfg-ntp | Specify NTP server

-a Set the NIC to auto negotiate
-d half¦full Duplex options
-s Speed for NIC (val is 10,100,1000,10000)
-l Lists the NICS
-L Link
-A Add
-D Delete a port group
-d Delete a virtual switch
-U Remove an uplink from a vSwitch (unlink)
-Q Remove an uplink from a dvSwitch
-IP Assign IP address
-m Set MTU
-n Set netmask

Create a virtual switch
vicfg-vswitch  -a
Add a port group to a virtual switch
vicfg-vswitch  -A
Link an uplink to a virtual switch
vicfg-vswitch –L
Link an uplink to a distributed vSwitch
vicfg-vswitch –P  -v
Set Max MTU size
vicfg-vswitch  -m 9000
vicfg-vswitch –v  -P
Remove a VLAN ID
vicfg-vswitch  -v 0 –P

More information on the vMA can be found in the  vMA guide
Additional commands can be found over at Ray Heffer’s blog