Archive for : September, 2010


Understanding virtual machine memory

Memory in a virtualised environment is split up in three memory types.

Virtual Memory – Allocates memory through a syscall to the operating system.  This runs at the application level in the same way on virtual machines as in physical machines

Physical Memory – Runs at the OS level.  In simplistic terms it uses an ‘allocated’ and a ‘free’ list.  The application asks for memory from the physical memory so the OS puts moves the memory from the free list to the allocated list.

Machine Memory – The actual physically installed memory, runs at the Hypervisor level.

VM memory allocation starts with no memory then the hypervisor allocates machine memory to the physical memory.  When this memory gets released the free memory remains on the OS and it isn’t returned to the hypervisor.

A more in depth way of looking at this is that the memory is generally administered by what is known as software based memory virtualisation.
Each virtual machine’s (VM) memory is controlled by the virtual memory manager. (VMM)
The VMM for each VM maintains a mapping from the guest OS’s memory pages called the physical pages to the physical memory pages of the underlying host machine, called the machine pages.

Each VM sees a contiguous zero-based addressable memory space, however the underlying machine memory may not be contiguous, as it may be running more than one VM at a time.
The VMM intercepts virtual machine instructions that manipulate guest operating system memory management structures so that the actual memory management unit (MMU) on the processor is not updated directly by the virtual machine.
The ESX/ESXi host maintains the virtual-to-machine page mappings in a shadow page table that is kept up to date with the physical-to-machine mappings (maintained by the VMM).

The processor uses the translation lookaside buffer (TLB) on the processor cache for the direct virtual-to-physical machine mappings.
Some CPU’s support hardware assisted memory virtualisation
AMD SVM-V and Intel Xeon 5500 series support it. These CPU’s have two paging tables;
one for the virtual-to-physical translations
one for the physical-to-machine translations
Although hardware assisted memory virtualisation eliminates the overhead associated with software virtualisation, namely the overhead associated with keeping shadow page tables synchronised with guest page tables the TLB miss latency is significantly higher. As a result workloads with a small amount of page table activity will not have a detrimental effect using software virtualisation, whereas workloads with a lot of page table activity are likely to benefit from hardware assistance.

Transparent memory sharing
Transparent memory is on-the-fly de-duplication of memory by looking for identical copies of memory and deleting all but one copy, giving the impression that more memory is available to the virtual machine. You can set the rate with Mem.ShareScanTime and MemShareGhz in the advanced options.
Disable with Sched.mem.Pshare.enable set to false.
Use resxtop and esxtop to view PSHARE field of the interactive mode in the memory view.

Virtual machine memory in the vSphere client
Performance tab
Memory Granted – Amount of physical memory mapped to machine memory
Memory Shared – Amount of physical memory whose mapped machine memory has multiple pieces of phyiscal memory mapped to it.
Memory Consumed – Amount of machine memory that has physical memory mapped to it.
Memory Shared Common – Amount of machine memory with multiple pieces of physical memory mapped to it.

Resource Pools
When creating resource pools the system uses admission control to make sure that you cannot allocate what isn’t available.

Resources are considered reserved regardless of whether VM’s are associated with the pool or not.

Expandable Reservations
The pool can use resources from it’s parent or ancestors if this check box is selected.

When you move a VM into a resource pool its existing reservation and limits do not change. However shares will reflect the total number of shares in the new resource pool.

Reservation types
Fixed – A predefined limit is imposed and cannot be exceeded
Expandable (default) – Resource pool can borrow resources from its parent resource pool if the parent resource pool has expandable reservation ticked. This setting doesn’t allow you to exceed reservations or limit settings imposed.

Futher information can be found in the vSphere resource management guide.

Virtual Machine configuration and maximums

Virtual machines are made up of the following files.
vmname.vmx – Config file
vmname.vmdk – Describes charateristics
vmname-flat.vmdk – (hidden by default) Contains the data
vmname.nvram – VM BIOS
vmname.log – log file
vmware#.log – VMware log file
vmname.vswp – Virtual machine swap file on the ESX(i) host
vmname.vmsd – snapshot descriptor file

Limits per VM
Can have up to 8 CPU’s
2TB-512B Disk size
4 IDE devices (1 controller)
2 FDD (1 controller)
10 vNICs
20 USB devices (1 controller)
3 parallel devices
4 serial devices
4 SCSI adapters
15 SCSI targets per adapter
60 SCSI targets per VM
40 concurrent remote console connections per VM

VM vNIC types
vlance – AKA PCNet32. Supported by most 32 bit OS’s. 100MB max
vmxnet – Better performance
flexible – Either one of the above
e1000 – High performance
enhanced vmxnet – Enhanced performance, includes support for jumbo frames
vmxnet3 – Builds on vmxnet. Allows you to scale TCP/IP traffic flow in Windows Server 2008 (receive size scalability)

Paravirtualization allows the VM to talk direct to the lower levels of the virtual machine for better performance

Virtual machine hardware types 4 & 7; 7 is required for older OS support and newer networking types