Features/High Availability Container Resources

From FedoraProject

< Features(Difference between revisions)
Jump to: navigation, search
m (Scope)
(Release Notes)
 
(34 intermediate revisions by 2 users not shown)
Line 8: Line 8:
 
== Summary ==
 
== Summary ==
 
<!-- A sentence or two summarizing what this feature is and what it will do.  This information is used for the overall feature summary page for each release. -->
 
<!-- A sentence or two summarizing what this feature is and what it will do.  This information is used for the overall feature summary page for each release. -->
The Container Resources feature allows the HA stack (Pacemaker + Corosync) residing on a host machine to extend management of resources into virtual guest instances (KVM/LXC).
+
The Container Resource feature allows the HA stack (Pacemaker + Corosync) residing on a host machine to extend management of resources into virtual guest instances (KVM/Linux Containers) through use of the pacemaker_remote service.
  
 
== Owner ==
 
== Owner ==
Line 19: Line 19:
 
== Current status ==
 
== Current status ==
 
* Targeted release: [[Releases/19 | Fedora 19 ]]  
 
* Targeted release: [[Releases/19 | Fedora 19 ]]  
* Last updated: 1-25-2013
+
* Last updated: 2013-03-12
* Percentage of completion: 60%
+
* Percentage of completion: 100%
  
 
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
 
<!-- CHANGE THE "FedoraVersion" TEMPLATES ABOVE TO PLAIN NUMBERS WHEN YOU COMPLETE YOUR PAGE. -->
Line 28: Line 28:
 
This feature is in response to the growing desire for high availability functionality to be extended outside of the host into virtual guest instances.  Pacemaker is currently capable of managing virtual guests, meaning Pacemaker can start/stop/monitor/migrate virtual guests anywhere in the cluster, but Pacemaker has no ability to manage the resources that live within the virtual guests.  At the moment these virtual guests are very much a black box to Pacemaker.
 
This feature is in response to the growing desire for high availability functionality to be extended outside of the host into virtual guest instances.  Pacemaker is currently capable of managing virtual guests, meaning Pacemaker can start/stop/monitor/migrate virtual guests anywhere in the cluster, but Pacemaker has no ability to manage the resources that live within the virtual guests.  At the moment these virtual guests are very much a black box to Pacemaker.
  
The Container Resources feature changes this by giving Pacemaker the ability to reach into the virtual guests and manage resources in the exact same way resources are managed on the host nodes. Ultimately this gives the HA stack the ability to manage resources across all the nodes in cluster as well as any virtual guests that reside within those cluster nodes.
+
The Container Resources feature changes this by giving Pacemaker the ability to reach into the virtual guests and manage resources in the exact same way resources are managed on the host nodes. Ultimately this gives the HA stack the ability to manage resources across all the nodes in cluster as well as any virtual guests that reside within those cluster nodes.  The key component that makes this functionality possible is the new pacemaker_remote service which allows pacemaker to integrate non-cluster nodes (remote-nodes not running corosynce+pacemaker) into the cluster.
  
 
[[File:Ha node without container resources.jpg]]
 
[[File:Ha node without container resources.jpg]]
Line 42: Line 42:
 
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
 
<!-- What work do the developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
  
'''Remote-LRMD:'''
+
'''pacemaker_remote:'''
  
Pacemaker's existing LRMD must be modified to allow the client and server to communicate remotely over tcp.  TLS will be used with PSK encryption/authentication to secure the connection between the LRMD client and server.  A standalone version of the LRMD running a tls backend on a virtual guest will allow the CRMD on a host machine to manage resources within the virtual guest.
+
The pacemaker_remote service is a modified version of Pacemaker's existing LRMD that allows for remote communication over tcp.  This service also provides an interface for remote nodes to communicate with the connecting cluster node's internal components (cib, crmd, attrd, stonithd).  TLS is used with PSK encryption/authentication to secure the connection between the pacemaker_remote's client and server.  Cluster nodes integrate remote-nodes into the cluster by connecting to the remote-node's pacemaker_remote service.
  
 
'''Pengine Container Resource Support:'''
 
'''Pengine Container Resource Support:'''
  
Pacemaker's policy engine component needs to be able to understand how to represent and manage container resources.  This is as simple as the policy engine understanding that container resources are both a resource and a location other resources can be placed after the container resource has started.  The policy engine will contain routing information in the LRMD action specify which LRMD an action should go to and on what node.  This will allow the CRMD to route the actions to a remote LRMD instance living on the virtual guest rather than the local LRMD instance living on the host node.
+
Pacemaker's policy engine component needs to be able to understand how to represent and manage container resources.  This is as simple as the policy engine understanding that container resources are both a resource and a location other resources can be placed after the container resource has started.  The policy engine will contain routing information in each resource action to specify which node (whether it be the local host or remote guest) an action should go to.  This is how the CRMD will know to route certain actions to a pacemaker_remote instance living on a virtual guest rather than the local LRMD instance living on the host.
  
 
'''CRMD-Routing:'''
 
'''CRMD-Routing:'''
  
Pacemaker's CRMD component must now be capable of routing LRMD commands to both the local LRMD and remote LRMD instances residing on virtual guests.
+
Pacemaker's CRMD component must now be capable of routing LRMD commands to both the local LRMD and pacemaker_remote instances residing on virtual guests.
  
'''LXC Resource Agent:'''
+
'''Defining Linux Container Use-cases:'''
  
A fedora supported LXC resource agent must be created.  It is likely that there is already open source community support for such an agent somewhere that this can be based off.
+
Part of this project is to identify interesting use-cases for Linux Containers in an HA environment.  This project aims at exploring use of libvirt-lxc in HA environments to dynamically define and launch Linux Containers throughout the cluster as well as manage the resources within those containers.
  
 
'''PCS management support for Container Resources:'''
 
'''PCS management support for Container Resources:'''
Line 76: Line 76:
 
3. What are the expected results of those actions?
 
3. What are the expected results of those actions?
 
-->
 
-->
 +
 +
These are the high-level steps required to use this feature with kvm guests using libvirt.  It is also possible to use this with libvirt-lxc guests, but since the required steps for getting this to work with linux containers are not as generic as the kvm case, this will focus on solely the kvm case for now.
 +
 +
1. Generate a pacemaker_remote authentication key.
 +
 +
* dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1
 +
 +
Place the authkey file in the /etc/pacemaker/ folder on every cluster-node that is capable of running launching a kvm guest.
 +
 +
2. On a pacemaker cluster-node create virtual kvm guest running fedora 19.
 +
* Define a static network address and make sure to set the hostname.
 +
* Install the pacemaker-remote and resoure-agents packages.  Set pacemaker-remote to launch on startup (systemclt enable pacemaker_remote).
 +
* Install the authkey created in step 1 in the /etc/pacemaker/ folder.
 +
 +
3. Using pcs on cluster-node, define virtual guest resource using the VirtualDomain agent.
 +
 +
In this example the resource is called 'vm-guest1' which launches the kvm guest created in step 2.  After the cluster node launches vm-guest1, it attempts to connect to that node's pacemaker_remote service using the hostname supplied in the remote-node meta-attribute.  In this case pacemaker expects to be able to contact the hostname 'guest1' after the 'vm-guest1' resource launches.  Once that connection is made, 'guest1' will become visible as a node capable of running resources when viewing the output of crm_mon.
 +
 +
Make sure that the resource id and the remote-node hostname are unique to one another. They cannot be the same.  Below is an example of how to configure vm-guest1 as a Container Resource using pcs.
 +
 +
* pcs resource create vm-guest1 VirtualDomain hypervisor="qemu:///system" config="vm-guest1.xml" meta remote-node=guest1
 +
 +
4. Using pcs on host node, define Dummy resource.
 +
 +
* pcs resource create FAKE Dummy
 +
 +
Run crm_mon to see where pacemaker launched the 'FAKE' resource.  To force the resource to run on the 'guest1' node run this.
 +
 +
* pcs constraint location FAKE perfers guest1
 +
 +
5. The kvm guest should now be able to run resources as just as if it was a cluster-node.
  
 
== User Experience ==
 
== User Experience ==
 
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
 
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
 +
 +
Users will be able to define resources as container resources.  Once a container resource is defined, that resource becomes a location capable of running resources just as if it was another cluster node.
  
 
== Dependencies ==
 
== Dependencies ==
 
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
 
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
 +
 +
No new Pacemaker dependencies will be required for this feature. A list of the current dependencies can be found here. [https://github.com/ClusterLabs/pacemaker]
  
 
== Contingency Plan ==
 
== Contingency Plan ==
 
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
 
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not.  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
 +
 +
The part of the project surrounding researching and defining use-cases for Linux Containers in an HA environment runs the risk of being incomplete before Fedora 19.  The Linux Container support required to achieve this goal is not well understood and may or may not be possible depending on the capabilities of libvirt-lxc and potentially systemd at the time of release.  This outcome will not affect the release this feature however as the KVM use-case is well understood.
 +
 +
If this feature is not complete by development freeze, the functionality necessary to configure this feature will be disabled in the stable configuration scheme.  The feature is designed in such a way that it should be able to be disabled without without negatively affecting any existing functionality.
  
 
== Documentation ==
 
== Documentation ==
 
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
 
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
*
+
* Upstream documentation specific to this feature is complete and lives here, [http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Remote/index.html Pacemaker Remote Doc]
  
 
== Release Notes ==
 
== Release Notes ==
 
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
 
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
 
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
 
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
*
+
* Pacemaker now supports the ability to manage resources remotely on non-cluster nodes through the use of the pacemaker_remote service.  This feature allows pacemaker to manage both virtual guests and the resources that live within the guests all from the host cluster node without requiring the guest nodes to run the cluster stack. View http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Remote for more information.
  
 
== Comments and Discussion ==
 
== Comments and Discussion ==
* See [[Talk:Features/Your_Feature_Name]]  <!-- This adds a link to the "discussion" tab associated with your page.  This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
+
* See [[Talk:Features/High_Availability_Container_Resources]]  <!-- This adds a link to the "discussion" tab associated with your page.  This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
  
  
[[Category:FeaturePageIncomplete]]
+
[[Category:FeatureAcceptedF19]]
 
<!-- When your feature page is completed and ready for review -->
 
<!-- When your feature page is completed and ready for review -->
 
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
 
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
 
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
 
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
 
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->
 
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Latest revision as of 22:37, 13 May 2013


Contents

[edit] High Availability Container Resources

[edit] Summary

The Container Resource feature allows the HA stack (Pacemaker + Corosync) residing on a host machine to extend management of resources into virtual guest instances (KVM/Linux Containers) through use of the pacemaker_remote service.

[edit] Owner

  • Email: <dvossel@redhat.com>

[edit] Current status

  • Targeted release: Fedora 19
  • Last updated: 2013-03-12
  • Percentage of completion: 100%


[edit] Detailed Description

This feature is in response to the growing desire for high availability functionality to be extended outside of the host into virtual guest instances. Pacemaker is currently capable of managing virtual guests, meaning Pacemaker can start/stop/monitor/migrate virtual guests anywhere in the cluster, but Pacemaker has no ability to manage the resources that live within the virtual guests. At the moment these virtual guests are very much a black box to Pacemaker.

The Container Resources feature changes this by giving Pacemaker the ability to reach into the virtual guests and manage resources in the exact same way resources are managed on the host nodes. Ultimately this gives the HA stack the ability to manage resources across all the nodes in cluster as well as any virtual guests that reside within those cluster nodes. The key component that makes this functionality possible is the new pacemaker_remote service which allows pacemaker to integrate non-cluster nodes (remote-nodes not running corosynce+pacemaker) into the cluster.

Ha node without container resources.jpg

Ha node w container resources.jpg

[edit] Benefit to Fedora

This feature expands our current high availability functionality to span across both physical bare-metal cluster nodes and the virtual environments that reside within those nodes. Without this feature, there is currently no direct approach for achieving this functionality.

[edit] Scope

pacemaker_remote:

The pacemaker_remote service is a modified version of Pacemaker's existing LRMD that allows for remote communication over tcp. This service also provides an interface for remote nodes to communicate with the connecting cluster node's internal components (cib, crmd, attrd, stonithd). TLS is used with PSK encryption/authentication to secure the connection between the pacemaker_remote's client and server. Cluster nodes integrate remote-nodes into the cluster by connecting to the remote-node's pacemaker_remote service.

Pengine Container Resource Support:

Pacemaker's policy engine component needs to be able to understand how to represent and manage container resources. This is as simple as the policy engine understanding that container resources are both a resource and a location other resources can be placed after the container resource has started. The policy engine will contain routing information in each resource action to specify which node (whether it be the local host or remote guest) an action should go to. This is how the CRMD will know to route certain actions to a pacemaker_remote instance living on a virtual guest rather than the local LRMD instance living on the host.

CRMD-Routing:

Pacemaker's CRMD component must now be capable of routing LRMD commands to both the local LRMD and pacemaker_remote instances residing on virtual guests.

Defining Linux Container Use-cases:

Part of this project is to identify interesting use-cases for Linux Containers in an HA environment. This project aims at exploring use of libvirt-lxc in HA environments to dynamically define and launch Linux Containers throughout the cluster as well as manage the resources within those containers.

PCS management support for Container Resources:

The pcs management tool for pacemaker needs to support whatever configuration mechanism is settled upon for representing container resources in the CIB.

[edit] How To Test

These are the high-level steps required to use this feature with kvm guests using libvirt. It is also possible to use this with libvirt-lxc guests, but since the required steps for getting this to work with linux containers are not as generic as the kvm case, this will focus on solely the kvm case for now.

1. Generate a pacemaker_remote authentication key.

  • dd if=/dev/urandom of=/etc/pacemaker/authkey bs=4096 count=1

Place the authkey file in the /etc/pacemaker/ folder on every cluster-node that is capable of running launching a kvm guest.

2. On a pacemaker cluster-node create virtual kvm guest running fedora 19.

  • Define a static network address and make sure to set the hostname.
  • Install the pacemaker-remote and resoure-agents packages. Set pacemaker-remote to launch on startup (systemclt enable pacemaker_remote).
  • Install the authkey created in step 1 in the /etc/pacemaker/ folder.

3. Using pcs on cluster-node, define virtual guest resource using the VirtualDomain agent.

In this example the resource is called 'vm-guest1' which launches the kvm guest created in step 2. After the cluster node launches vm-guest1, it attempts to connect to that node's pacemaker_remote service using the hostname supplied in the remote-node meta-attribute. In this case pacemaker expects to be able to contact the hostname 'guest1' after the 'vm-guest1' resource launches. Once that connection is made, 'guest1' will become visible as a node capable of running resources when viewing the output of crm_mon.

Make sure that the resource id and the remote-node hostname are unique to one another. They cannot be the same. Below is an example of how to configure vm-guest1 as a Container Resource using pcs.

  • pcs resource create vm-guest1 VirtualDomain hypervisor="qemu:///system" config="vm-guest1.xml" meta remote-node=guest1

4. Using pcs on host node, define Dummy resource.

  • pcs resource create FAKE Dummy

Run crm_mon to see where pacemaker launched the 'FAKE' resource. To force the resource to run on the 'guest1' node run this.

  • pcs constraint location FAKE perfers guest1

5. The kvm guest should now be able to run resources as just as if it was a cluster-node.

[edit] User Experience

Users will be able to define resources as container resources. Once a container resource is defined, that resource becomes a location capable of running resources just as if it was another cluster node.

[edit] Dependencies

No new Pacemaker dependencies will be required for this feature. A list of the current dependencies can be found here. [1]

[edit] Contingency Plan

The part of the project surrounding researching and defining use-cases for Linux Containers in an HA environment runs the risk of being incomplete before Fedora 19. The Linux Container support required to achieve this goal is not well understood and may or may not be possible depending on the capabilities of libvirt-lxc and potentially systemd at the time of release. This outcome will not affect the release this feature however as the KVM use-case is well understood.

If this feature is not complete by development freeze, the functionality necessary to configure this feature will be disabled in the stable configuration scheme. The feature is designed in such a way that it should be able to be disabled without without negatively affecting any existing functionality.

[edit] Documentation

[edit] Release Notes

  • Pacemaker now supports the ability to manage resources remotely on non-cluster nodes through the use of the pacemaker_remote service. This feature allows pacemaker to manage both virtual guests and the resources that live within the guests all from the host cluster node without requiring the guest nodes to run the cluster stack. View http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Remote for more information.

[edit] Comments and Discussion