When compared to a bare metal physical server which often have over-provisioned resources of CPU, memory and hard disk storage, resulting in lots of wastage of resources, a VM can be right-sized to fit the application running on it, thereby freeing up resources for other VMs and applications. However the ESXi host is still going to run out of resources at some point of time when you have VMs requesting more resources than what the ESXi host is capable of providing.
We can ensure that a VM's guest OS and its applications get the resources they need without being deprived by other guest OSs and their applications. These controls are referred to as Reservations, Limits and Shares.
Reservations: Reservations help ensure a certain minimal amount of resources available to the VM irrespective of what else and what other applications are running on the ESXi host.
Limits: Limits place a threshold on the given resource a VM can use. This feature is quite granular in how resources are utilized. Depending on the resource to which the limit is being applied, the ESXi host behavior will vary.
Shares: Shares help prioritize resources during periods of resource contention. When VMs are competing for scarce resources, the ESXi host decides which VM gets to use certain resources depending on the Shares configurations. VMs with higher shares allocated will get higher priority to the ESXi's host resources.
Let us say you start creating more VMs on the ESXi host and allocate memoryto them. At some point of time, you will have allocated more memory to the VMs than whatever is available on the ESXi host. VMware ESXi supports a number of technologies that do advanced memory management:
Idle Memory Tax
Even before VMware ESXi aggressively starts making enhancements to relieve memory demand, it ensures that VMs do not pile up or stock memory unnecessarily by "charging" more for the idle memory. Up to 75% of the memory allocated to a VM can be borrowed to service another VM by Idle Memory Tax (IDT). This is a configurable parameter. Usually it is not necessary to do this and is not recommended. Inside the VM guest OS, VMware Tools will expand its balloon driver to figure out which memory blocks are allocated but idle and therefore available to be used somewhere else.
Transparent Page Sharing (TPS)
Transparent Page Sharing (TPS) is a mechanism used by ESXi in which identical memory pages are shared amongst VMs to reduce the net count of memory pages consumed. The ESXi hypervisor computes hashes of the contents of memory pages to identify and mark pages that contain identical memory data. If it finds a hash match, it compares the matching memory pages to exclude a false positive. Once the pages are verified to be identical, the hypervisor will transparently remap the memory pages of the VMs so that they are sharing the same physical memory page, thus reducing overall memory consumption. Some advanced parameters are available to finetune the behavior of the page sharing mechanisms.
VMware indicated that TPS is no longer enabled by default. This is because of a research paper that demonstrates using TPS to gain access to the AES encryption key of a machine sharing pages. This is a security risk, therefore VMware made the decision to disable TPS by default and leaves the onus on customers to evaluate the risk of enabling TPS in their environment if they so desire.
Ballooning
Ballooning involves the use of a driver installed into the VM guest OS. This driver gets installed when you install VMware Tools. This balloon driver can respond to commands from the ESXi hypervisor to reclaim memory from that particular guest OS. The balloon driver accomplishes this by requesting memory from the guest OS, a process called inflating, and then passing that memory back to the hypervisor to be used by other VMs.
Usually when the ESXi reclaims memory from the VM through the balloon driver there is no performance impact on the guest OS since this memory was not being used anyway. However in case the amount of memory configured for that VM is already insufficient for the guest OS and its applications, it's quite a possibility that inflating the balloon driver will invoke the guest OS paging or swapping, affecting the VM performance.
The balloon driver is OS-specific, so it's different for Linux and Windows. Comes as part of the VMware Tools. When the ESXi host is running low on memory, it i.e. the hypervisor will signal the balloon driver to grow. The balloon driver will request memory from the OS. This causes the balloon driver's memory footprint to grow or inflate. The memory that is passed to the balloon driver is then given to the hypervisor. The hypervisor uses this memory to be granted to other VMs on the host, reducing the swap activity and minimizing the performance impact due to memory constraints. When the memory demand on the host decreases, the balloon driver will deflate, and return memory to the VM.
In certain cases, inflating the balloon driver can release memory back to the hypervisor without diminishing the VM performance because the guest OS can give the balloon driver unused or idle pages.
Memory Compression
When an ESXi host gets to a point where hypervisor swapping becomes necessary, the VMkernel will attempt to compress memory pages and keep them in RAM in a compressed memory cache. Pages that can be compressed by at least 50% are put into the compressed memory cache instead of being written to disk and can then be recovered much more quickly if the guest OS needs that memory page, since memory access is a million times faster than disk access. Memory compression can drastically reduce the number of pages that must be swapped to disk and can thus dramatically improve the performance of an ESXi host that is under strong memory pressure. A configurable amount of VM memory is used for the compression cache, 10% by default, but this starts at zero and grows as needed when VM memory starts swapping out. Compression is invoked only when the ESXi host reaches the point when VMkernel swapping is needed.
Swapping
We have two types of swapping when memory is managed by VMware ESXi. The first type is guest OS swapping in which the guest OS inside the VM swaps the pages out to its virtual disk due to its own memory management algorithms. This is generally due to memory requirements that are greater than available memory. This is the scenario when a VM is configured with less memory than the guest OS and its applications require. Guest OS is controlled completely by the guest OS, and the hypervisor has no hold over that aspect.
The second type of swapping is hypervisor swapping. In case none of the previously described methodologies trim the guest OS memory usage sufficiently, the ESXi host is forced to resort to use hypervisor swapping. Hypervisor swapping involves the ESXi swapping memory pages out to disk in order to reclaim memory that is needed somewhere else. ESXi hypervisor swapping happens without any regard to whether the pages are being actively used by the VM guest OS. As a consequence, and because disk response times is millions of times slower than memory response times, guest OS performance is severely impacted if hypervisor swapping is invoked. It is for this rationale that ESXi will not invoke swapping unless and until it is absolutely necessary, if you have run out of options after having tried all other memory management techniques.
Another vital point to note is that you should avoid hypervisor swapping under all except the most extraneous circumstances; there is a significant and noticeable impact to performance. Even swapping to SSDs is considerably slower than directly accessing RAM.
We can ensure that a VM's guest OS and its applications get the resources they need without being deprived by other guest OSs and their applications. These controls are referred to as Reservations, Limits and Shares.
Reservations: Reservations help ensure a certain minimal amount of resources available to the VM irrespective of what else and what other applications are running on the ESXi host.
Limits: Limits place a threshold on the given resource a VM can use. This feature is quite granular in how resources are utilized. Depending on the resource to which the limit is being applied, the ESXi host behavior will vary.
Shares: Shares help prioritize resources during periods of resource contention. When VMs are competing for scarce resources, the ESXi host decides which VM gets to use certain resources depending on the Shares configurations. VMs with higher shares allocated will get higher priority to the ESXi's host resources.
Let us say you start creating more VMs on the ESXi host and allocate memoryto them. At some point of time, you will have allocated more memory to the VMs than whatever is available on the ESXi host. VMware ESXi supports a number of technologies that do advanced memory management:
Idle Memory Tax
Even before VMware ESXi aggressively starts making enhancements to relieve memory demand, it ensures that VMs do not pile up or stock memory unnecessarily by "charging" more for the idle memory. Up to 75% of the memory allocated to a VM can be borrowed to service another VM by Idle Memory Tax (IDT). This is a configurable parameter. Usually it is not necessary to do this and is not recommended. Inside the VM guest OS, VMware Tools will expand its balloon driver to figure out which memory blocks are allocated but idle and therefore available to be used somewhere else.
Transparent Page Sharing (TPS)
Transparent Page Sharing (TPS) is a mechanism used by ESXi in which identical memory pages are shared amongst VMs to reduce the net count of memory pages consumed. The ESXi hypervisor computes hashes of the contents of memory pages to identify and mark pages that contain identical memory data. If it finds a hash match, it compares the matching memory pages to exclude a false positive. Once the pages are verified to be identical, the hypervisor will transparently remap the memory pages of the VMs so that they are sharing the same physical memory page, thus reducing overall memory consumption. Some advanced parameters are available to finetune the behavior of the page sharing mechanisms.
VMware indicated that TPS is no longer enabled by default. This is because of a research paper that demonstrates using TPS to gain access to the AES encryption key of a machine sharing pages. This is a security risk, therefore VMware made the decision to disable TPS by default and leaves the onus on customers to evaluate the risk of enabling TPS in their environment if they so desire.
Ballooning
Ballooning involves the use of a driver installed into the VM guest OS. This driver gets installed when you install VMware Tools. This balloon driver can respond to commands from the ESXi hypervisor to reclaim memory from that particular guest OS. The balloon driver accomplishes this by requesting memory from the guest OS, a process called inflating, and then passing that memory back to the hypervisor to be used by other VMs.
Usually when the ESXi reclaims memory from the VM through the balloon driver there is no performance impact on the guest OS since this memory was not being used anyway. However in case the amount of memory configured for that VM is already insufficient for the guest OS and its applications, it's quite a possibility that inflating the balloon driver will invoke the guest OS paging or swapping, affecting the VM performance.
The balloon driver is OS-specific, so it's different for Linux and Windows. Comes as part of the VMware Tools. When the ESXi host is running low on memory, it i.e. the hypervisor will signal the balloon driver to grow. The balloon driver will request memory from the OS. This causes the balloon driver's memory footprint to grow or inflate. The memory that is passed to the balloon driver is then given to the hypervisor. The hypervisor uses this memory to be granted to other VMs on the host, reducing the swap activity and minimizing the performance impact due to memory constraints. When the memory demand on the host decreases, the balloon driver will deflate, and return memory to the VM.
In certain cases, inflating the balloon driver can release memory back to the hypervisor without diminishing the VM performance because the guest OS can give the balloon driver unused or idle pages.
Memory Compression
When an ESXi host gets to a point where hypervisor swapping becomes necessary, the VMkernel will attempt to compress memory pages and keep them in RAM in a compressed memory cache. Pages that can be compressed by at least 50% are put into the compressed memory cache instead of being written to disk and can then be recovered much more quickly if the guest OS needs that memory page, since memory access is a million times faster than disk access. Memory compression can drastically reduce the number of pages that must be swapped to disk and can thus dramatically improve the performance of an ESXi host that is under strong memory pressure. A configurable amount of VM memory is used for the compression cache, 10% by default, but this starts at zero and grows as needed when VM memory starts swapping out. Compression is invoked only when the ESXi host reaches the point when VMkernel swapping is needed.
Swapping
We have two types of swapping when memory is managed by VMware ESXi. The first type is guest OS swapping in which the guest OS inside the VM swaps the pages out to its virtual disk due to its own memory management algorithms. This is generally due to memory requirements that are greater than available memory. This is the scenario when a VM is configured with less memory than the guest OS and its applications require. Guest OS is controlled completely by the guest OS, and the hypervisor has no hold over that aspect.
The second type of swapping is hypervisor swapping. In case none of the previously described methodologies trim the guest OS memory usage sufficiently, the ESXi host is forced to resort to use hypervisor swapping. Hypervisor swapping involves the ESXi swapping memory pages out to disk in order to reclaim memory that is needed somewhere else. ESXi hypervisor swapping happens without any regard to whether the pages are being actively used by the VM guest OS. As a consequence, and because disk response times is millions of times slower than memory response times, guest OS performance is severely impacted if hypervisor swapping is invoked. It is for this rationale that ESXi will not invoke swapping unless and until it is absolutely necessary, if you have run out of options after having tried all other memory management techniques.
Another vital point to note is that you should avoid hypervisor swapping under all except the most extraneous circumstances; there is a significant and noticeable impact to performance. Even swapping to SSDs is considerably slower than directly accessing RAM.
No comments:
Post a Comment