Working with ESX(i) Log Files

11 Jul 2011 by Simon Greaves

Working with ESX(i) log files is important when troubleshooting issues within the virtual environment. You can view and search log files in ESX(i) and in vCenter Server using a few different methods.

Methods

Using the vSphere client
The direct console user interface (DCUI)
A web browser
A syslog or vMA appliance
An SSH connection to the host
PowerCLI using the Get-Log command
When using SSH, use the following commands to view and search the log files.
Use more to page through the log files one page at a time
Use tail to view the end of the log files
Use grep to search
Use pipe | to link commands together
Use pipe | to grep to search through files
Use cat to concatenate & use grep to search
Use find | print | grep <filename> to search for a file

Example
Cat hostd.log | grep <search variable> | more

vCenter log files

vCenter log files are in a vpxd-xx.log format where xx is a numerical value that increases when each log file is 5MB in size.

The log file numbers rotate when the vpxd service is started or when the log reaches 5MB in size.

The log files are located in the c:\programdata\VMware\VMware virtual center\logs

Other log files include
Vpxd-alert-x.log
Vpxd-profile-x.log

Esx logs

/var/log/VMkernel - VMkernel messages
/var/log/vmkwarning
/var/log/vmksummary
/var/log/vmware/hostd.log
/var/log/messages - service console
/var/log/vmware/vpx/vpxa.log - vSphere client agent
/var/log/aam/VMware__xxx.log - HA
/var/log/vmkiscsid.log - iSCSI
/var/log/boot-logs/sysboot.log - boot log

ESXi logs

/var/log/messages - combination of the VMkernel and vmkwarning
/var/log/vmware/hostd.log - host management service
/var/log/vmware/vpx/vpxa.log - vSphere client agent
/var/log/sysboot.log - boot log
/var/log/vmware/aam/vmware__xxx.log - HA

Log rotation

Within ESX(i), rotation for most log files is controlled by /etc/logrotate.conf
To view available options run man logrotate

With both ESX and ESX(i) hostd.log rotation is controlled with /etc/vmware/hostd/config.xml

vpxa.log rotation is controlled with /etc/opt/VMware/vpxa/vpxa.cfg

Should you wish to edit the rotation control files you can use Nano in ESX or vi in ESX and ESXi.

I will focus on vi as I am more familiar with it and with the arrival of vSphere 5 there will no longer be ESX, and as such no native support for Nano.

vi commands

a - append
i - insert
O/o - open new line - O is line above, o is line below
r - replace
: - search or save options
/ - search
wq - write and quit
x - delete individual characters
dd - delete line
$ - go to the end of the line
ESC - break out of current mode

Log bundles

Log bundles can be accessed through the VMware folder on the start menu, by clicking generate vCenter server log bundle. This runs the vc-support windows scripting file located at c:\program filesVMware\virtual infrastructure\virtual centrescript\svc-support.wsf and cscript.

You can also download it through the vSphere client and by connecting to the ESX(I) server using scp with Veeam FastSCP or WinSCP. To do this you have to enable tech support mode first.

An alternative way of generating log bundles is through the vm-support command run through an SSH connection to the COS or through the vMA.  Running vm-support will generate a tar compressed file.

Procedure

[root@esxhost]# /usr/bin/vm-support

With ESXi it is possible to place log files on shared storage.  To set this open the vSphere client connection to the host, click configuration>advanced settings>syslog select local and enter the path to the shared storage.  Enter the log file location as [datastorename]/logfiles/hostname.log.

vilogd

vilogd is a service that performs log collections.

You can manage it with the vilogger commands.  vilogger is used to enable and disable or configure the log collections with these commands.

To use vilogger, first ensure that vi-fastpass is enabled using vifp list server to list out the current vi-fastpass enabled servers, if no servers are listed use vifp addserver <servername> and vifptarget -s <servername> to add again.

Commands

vilogger enable
vilogger list
vilogger update policy
Control the vilogd service with etc/init.d/vmware-vilogd <start|stop|restart>
vilogger has several parameters available, an example of which are
--numrotation number of files to collect
--maxfilesize specified in MB
--collectionperiod how often to poll, specified in seconds

Example
vilogger enable --server <servername> --numrotation 20 --maxfilesize 10 --collectionperiod 10

This command will collect the following logs from the ESXi host
hostd.log
messages.log
vpxa.log

To scroll through the log files one page at a time use the more command.

Example
more hostd.log

Configure vMA as a Syslog Server

You can configure the vMA as a syslog receiver to collect log files from the ESX and ESXi server.  Run the commands listed below to configure.

vMA

sudo service rsyslog stop
sudo nano /etc/sysconfig/rsyslog
This will open nano so you can edit the following information
change SYSLOGD_OPTIONS="-m 0" to SYSLOGD_OPTIONS="-r -m 0"

Save and exit the file
sudo service rsyslog start
sudo iptables -I INPUT -i eth0 -p udp --dport 514 -j ACCEPT
sudo nano /etc/rc.local

Edit the file to add the iptables line below to the end of the rc.local file

iptables -I INPUT -i eth0 -p udp --dport 514 -j ACCEPT

ESX

To configure ESX to use vMA as a syslog server add the IP address of the vMA to the /etc/syslog.conf file.

vi /etc/syslog.conf
Add the following lines to the bottom of the file

Send all syslog traffic to vMA

*.* @<IP_Address_Of_vMA>

Open the firewall with
/usr/sbin/esxcfg-firewall -o 514,udp,out,syslog
Finally restart the syslog service with
sbin/services syslog restart

ESXi

Use the vSphere client by going to configuration>advanced>syslog settings and enter into the syslog.Remote.Hostname section the name of the vMA.

Alternatively assuming vi-fastpass is enabled run
vifptarget -s [<ESXihost>]
vicfg-syslog -s [<vMA>]
vifptarget -c

Comments are closed for this post.