Working with ESX(i) Log Files
Working with ESX(i) log files is important when troubleshooting issues within the virtual environment. You can view and search log files in ESX(i) and in vCenter Server using a few different methods.
Using the vSphere client
The direct console user interface (DCUI)
A web browser
A syslog or vMA appliance
An SSH connection to the host
PowerCLI using the Get-Log command
When using SSH, use the following commands to view and search the log files.
Use more to page through the log files one page at a time
Use tail to view the end of the log files
Use grep to search
Use pipe | to link commands together
Use pipe | to grep to search through files
Use cat to concatenate & use grep to search
Use find | print | grep filename to search for a file
Cat hostd.log | grep search variable | more
vCenter log files
vCenter log files are in a vpxd-xx.log format where xx is a numerical value that increases when each log file is 5MB in size.
The log file numbers rotate when the vpxd service is started or when the log reaches 5MB in size.
The log files are located in the c:programdataVMwareVMware virtual centerlogs
Other log files include
/var/log/VMkernel – VMkernel messages
/var/log/messages – service console
/var/log/vmware/vpx/vpxa.log – vSphere client agent
/var/log/vmkiscsid.log – iSCSI
/var/log/boot-logs/sysboot.log – boot log
/var/log/messages – combination of the VMkernel and vmkwarning
/var/log/vmware/hostd.log – host management service
/var/log/vmware/vpx/vpxa.log – vSphere client agent
/var/log/sysboot.log – boot log
/var/log/vmware/aam/vmware__xxx.log – HA
Within ESX(i), rotation for most log files is controlled by /etc/logrotate.conf
To view available options run man logrotate
With both ESX and ESX(i) hostd.log rotation is controlled with /etc/vmware/hostd/config.xml
Vpxa.log rotation is controlled with /etc/opt/VMware/vpxa/vpxa.cfg
Should you wish to edit the rotation control files you can use Nano in ESX or vi in ESX and ESXi.
I will focus on vi as I am more familiar with it and with the arrival of vSphere 5 there will no longer be ESX, and as such no native support for Nano.
a – append
i – insert
O/o – open new line – O is line above, o is line below
r – replace
: – search or save options
/ – search
wq – write and quit
x – delete individual characters
dd – delete line
$ – go to the end of the line
ESC – break out of current mode
Log bundles can be accessed through the VMware folder on the start menu, by clicking generate vCenter server log bundle. This runs the vc-support windows scripting file located at c:program filesVMwarevirtual infrastructurevirtual centrescriptsvc-support.wsf and cscript.
You can also download it through the vSphere client and by connecting to the ESX(I) server using scp with Veeam FastSCP or WinSCP. To do this you have to enable tech support mode first.
An alternative way of generating log bundles is through the vm-support command run through an SSH connection to the COS or through the vMA. Running vm-support will generate a tar compressed file.
With ESXi it is possible to place log files on shared storage. To set this open the vSphere client connection to the host, click configuration>advanced settings>syslog select local and enter the path to the shared storage. Enter the log file location as [datastorename]/logfiles/hostname.log.
vilogd is a service that performs log collections.
You can manage it with the vilogger commands. vilogger is used to enable and disable or configure the log collections with these commands.
To use vilogger, first ensure that vi-fastpass is enabled using vifp list server to list out the current vi-fastpass enabled servers, if no servers are listed use vifp addserver servername and vifptarget -s servername to add again.
vilogger update policy
Control the vilogd service with etc/init.d/vmware-vilogd start|stop|restart
vilogger has several parameters available, an example of which are
–numrotation number of files to collect
–maxfilesize specified in MB
–collectionperiod how often to poll, specified in seconds
vilogger enable –server servername –numrotation 20 –maxfilesize 10 –collectionperiod 10
This command will collect the following logs from the ESXi host
To scroll through the log files one page at a time use the more command.
Configure vMA as a Syslog Server
You can configure the vMA as a syslog receiver to collect log files from the ESX and ESXi server. Run the commands listed below to configure.
#sudo service rsyslog stop
#sudo nano /etc/sysconfig/rsyslog
This will open nano so you can edit the following information
change SYSLOGD_OPTIONS=”-m 0″ to SYSLOGD_OPTIONS=”-r -m 0″
Save and exit the file
#sudo service rsyslog start
#sudo iptables -I INPUT -i eth0 -p udp –dport 514 -j ACCEPT
#sudo nano /etc/rc.local
Edit the file to add the iptables line below to the end of the rc.local file
iptables -I INPUT -i eth0 -p udp –dport 514 -j ACCEPT
To configure ESX to use vMA as a syslog server add the IP address of the vMA to the /etc/syslog.conf file.
Add the following lines to the bottom of the file
# Send all syslog traffic to vMA
Open the firewall with
#/usr/sbin/esxcfg-firewall -o 514,udp,out,syslog
Finally restart the syslog service with
#sbin/services syslog restart
Use the vSphere client by going to configuration>advanced>syslog settings and enter into the syslog.Remote.Hostname section the name of the vMA.
Alternatively assuming vi-faspass is enabled run
#vifptarget -s [ESXihost]
#vicfg-syslog -s [vMA]