Features/KVM Huge Page Backed Memory

From FedoraProject

< Features(Difference between revisions)
Jump to: navigation, search
(john posted a patch)
(How To Test)
 
(12 intermediate revisions by 4 users not shown)
Line 7: Line 7:
 
== Owner ==
 
== Owner ==
  
* Name: [[User:Chrisw|Chris Wright]], [[User:JohnCooper|John Cooper]]
+
* [[User:Chrisw|Chris Wright]] -- chrisw redhat com
 +
* [[User:JohnCooper|John Cooper]] -- john.cooper redhat com
  
 
== Current status ==
 
== Current status ==
  
 
* Targeted release: [[Releases/12|Fedora 12]]
 
* Targeted release: [[Releases/12|Fedora 12]]
* Last updated: 2009-07-23
+
* Last updated: 2009-09-24
* Percentage of completion: 90%
+
* Percentage of completion: 100%
  
=== TODO ===
+
An [https://bugzilla.redhat.com/515741 SELinux issue in the kernel] still remains, preventing full sVirt protection being applied to huge pages.
 
+
# Ensure the libvirt patch is in 0.7.0
+
# Testing
+
  
 
=== Completed ===
 
=== Completed ===
Line 24: Line 22:
 
# KVM support for huge pages
 
# KVM support for huge pages
 
# libvirt [http://www.redhat.com/archives/libvir-list/2009-July/msg00699.html patch posted] for running guests with <code>-mem-path /dev/hugepages</code>
 
# libvirt [http://www.redhat.com/archives/libvir-list/2009-July/msg00699.html patch posted] for running guests with <code>-mem-path /dev/hugepages</code>
 +
# libvirt [http://www.redhat.com/archives/libvir-list/2009-July/msg00753.html patch-v2 posted] which detects the host's hugetlbfs mount point automatically.
 +
# Identify [https://bugzilla.redhat.com/515741 SELinux issue in the kernel]
 +
# Get the libvirt patch committed to libvirt-0.7.1
  
 
== Detailed Description ==
 
== Detailed Description ==
  
x86 CPUs usually address memory in 4kB pages, but they also have the ability to use huge pages (4MB on x86_63, 2MB on x86_64 and x86_32 PAE).
+
x86 CPUs usually address memory in 4kB pages, but they also have the ability to use huge pages (4MB on x86_32, 2MB on x86_64 and x86_32 PAE).
  
By using huge pages for a KVM guest's memory, less memory is used for page tables and CPU cache misses are reduced, thereby increasing performance.
+
By using huge pages for a KVM guest's memory, less memory is used for page tables and TLB misses are reduced, thereby increasing performance.
  
 
Using huge pages for guest memory does have a downside, however - you can no longer swap nor balloon guest memory.
 
Using huge pages for guest memory does have a downside, however - you can no longer swap nor balloon guest memory.
Line 36: Line 37:
  
 
# Mount hugetlbfs to <code>/dev/hugepages</code> - <code>mount -t hugetlbfs hugetlbfs /dev/hugepages</code>
 
# Mount hugetlbfs to <code>/dev/hugepages</code> - <code>mount -t hugetlbfs hugetlbfs /dev/hugepages</code>
# Reserve some memory for huge pages - <code>sctl vm.nr_hugepages=516</code>
+
# Reserve some memory for huge pages - <code>sysctl vm.nr_hugepages=516</code>
 
# Pass the hugetlbfs path to <code>qemu-kvm</code> - <code>qemu-kvm -mem-path /dev/hugepages</code>
 
# Pass the hugetlbfs path to <code>qemu-kvm</code> - <code>qemu-kvm -mem-path /dev/hugepages</code>
  
Line 52: Line 53:
  
 
# <code>mount -t hugetlbfs hugetlbfs /dev/hugepages</code>
 
# <code>mount -t hugetlbfs hugetlbfs /dev/hugepages</code>
# Reserve some memory for huge pages - <code>sctl vm.nr_hugepages=516</code>
+
# Reserve some memory for huge pages - <code>sysctl vm.nr_hugepages=516</code>
# Run an existing KVM guest with libvirt
+
# Run an existing KVM guest with libvirt with the following added to the guest config:
 +
: <memoryBacking>
 +
:    <hugepages/>
 +
:  </memoryBacking>
 
# Confirm the guest has booted correctly
 
# Confirm the guest has booted correctly
 
# Confirm the guest is using hugepages by checking <code>HugePages_Free</code> in </code>/proc/meminfo</code>
 
# Confirm the guest is using hugepages by checking <code>HugePages_Free</code> in </code>/proc/meminfo</code>
Line 59: Line 63:
 
== User Experience ==
 
== User Experience ==
  
Users of KVM guests using huge page backed memory should experience reduced memory usage and improved performance.
+
Users of KVM guests using huge page backed memory should experience improved performance with some savings in host memory consumption.  The performance benefit is workload dependent.  But we have measured benefits on the order of 20% performance improvement (esp. useful when the guest's workload includes an application which itself is using hugepages).
  
 
== Dependencies ==
 
== Dependencies ==
Line 75: Line 79:
 
== Release Notes ==
 
== Release Notes ==
  
KVM guests running on Fedora can now take advantage of huge page backed memory to and benefit from improved performance and reduced memory usage.
+
KVM guests running on Fedora can now take advantage of huge page backed memory and benefit from improved performance and reduced memory usage.
  
 
== Comments and Discussion ==
 
== Comments and Discussion ==
Line 83: Line 87:
 
[[Category:Virtualization|KVM Huge Page Backed Memory]]
 
[[Category:Virtualization|KVM Huge Page Backed Memory]]
  
[[Category:FeatureReadyForWrangler]]
+
[[Category:FeatureAcceptedF12]]
  
 
[[Category:F12_Virt_Features|KVM Huge Page Backed Memory]]
 
[[Category:F12_Virt_Features|KVM Huge Page Backed Memory]]

Latest revision as of 17:16, 29 October 2009

Contents

[edit] KVM Huge Page Backed Memory

[edit] Summary

Enable KVM guests to use huge page backed memory in order to reduce memory consumption and improve performance by reducing CPU cache pressure.

[edit] Owner

[edit] Current status

  • Targeted release: Fedora 12
  • Last updated: 2009-09-24
  • Percentage of completion: 100%

An SELinux issue in the kernel still remains, preventing full sVirt protection being applied to huge pages.

[edit] Completed

  1. KVM support for huge pages
  2. libvirt patch posted for running guests with -mem-path /dev/hugepages
  3. libvirt patch-v2 posted which detects the host's hugetlbfs mount point automatically.
  4. Identify SELinux issue in the kernel
  5. Get the libvirt patch committed to libvirt-0.7.1

[edit] Detailed Description

x86 CPUs usually address memory in 4kB pages, but they also have the ability to use huge pages (4MB on x86_32, 2MB on x86_64 and x86_32 PAE).

By using huge pages for a KVM guest's memory, less memory is used for page tables and TLB misses are reduced, thereby increasing performance.

Using huge pages for guest memory does have a downside, however - you can no longer swap nor balloon guest memory.

In order to use huge pages with KVM, one must do the following:

  1. Mount hugetlbfs to /dev/hugepages - mount -t hugetlbfs hugetlbfs /dev/hugepages
  2. Reserve some memory for huge pages - sysctl vm.nr_hugepages=516
  3. Pass the hugetlbfs path to qemu-kvm - qemu-kvm -mem-path /dev/hugepages

This feature aims to improve allow huge pages to be used with libvirt managed guests.

[edit] Benefit to Fedora

Enables Fedora KVM hosts to achieve better performance.

[edit] Scope

Huge pages support in KVM already exists, so the remaining work is confined to libvirt.

[edit] How To Test

  1. mount -t hugetlbfs hugetlbfs /dev/hugepages
  2. Reserve some memory for huge pages - sysctl vm.nr_hugepages=516
  3. Run an existing KVM guest with libvirt with the following added to the guest config:
<memoryBacking>
<hugepages/>
</memoryBacking>
  1. Confirm the guest has booted correctly
  2. Confirm the guest is using hugepages by checking HugePages_Free in </code>/proc/meminfo</code>

[edit] User Experience

Users of KVM guests using huge page backed memory should experience improved performance with some savings in host memory consumption. The performance benefit is workload dependent. But we have measured benefits on the order of 20% performance improvement (esp. useful when the guest's workload includes an application which itself is using hugepages).

[edit] Dependencies

None, the work is confined to libvirt.

[edit] Contingency Plan

None needed.

[edit] Documentation

[edit] Release Notes

KVM guests running on Fedora can now take advantage of huge page backed memory and benefit from improved performance and reduced memory usage.

[edit] Comments and Discussion