High Availability Container Resources
The Container Resources feature allows the HA stack (Pacemaker + Corosync) residing on a host machine to extend management of resources into virtual guest instances (KVM/Linux Containers).
- Name: David Vossel
- Email: <email@example.com>
- Targeted release: Fedora 19
- Last updated: 2013-01-25
- Percentage of completion: 90%
This feature is in response to the growing desire for high availability functionality to be extended outside of the host into virtual guest instances. Pacemaker is currently capable of managing virtual guests, meaning Pacemaker can start/stop/monitor/migrate virtual guests anywhere in the cluster, but Pacemaker has no ability to manage the resources that live within the virtual guests. At the moment these virtual guests are very much a black box to Pacemaker.
The Container Resources feature changes this by giving Pacemaker the ability to reach into the virtual guests and manage resources in the exact same way resources are managed on the host nodes. Ultimately this gives the HA stack the ability to manage resources across all the nodes in cluster as well as any virtual guests that reside within those cluster nodes. The key component that makes this functionality possible is the new pacemaker_remote service which allows pacemaker to integrate non-cluster nodes (remote-nodes not running corosynce+pacemaker) into the cluster.
Benefit to Fedora
This feature expands our current high availability functionality to span across both physical bare-metal cluster nodes and the virtual environments that reside within those nodes. Without this feature, there is currently no direct approach for achieving this functionality.
The pacemaker_remote service is a modified version of Pacemaker's existing LRMD that allows for remote communication over tcp. This service also provides an interface for remote nodes to communicate with the connecting cluster node's internal components (cib, crmd, attrd, stonithd). TLS is used with PSK encryption/authentication to secure the connection between the pacemaker_remote's client and server. Cluster nodes integrate remote-nodes into the cluster by connecting to the remote-node's pacemaker_remote service.
Pengine Container Resource Support:
Pacemaker's policy engine component needs to be able to understand how to represent and manage container resources. This is as simple as the policy engine understanding that container resources are both a resource and a location other resources can be placed after the container resource has started. The policy engine will contain routing information in each resource action to specify which node (whether it be the local host or remote guest) an action should go to. This is how the CRMD will know to route certain actions to a pacemaker_remote instance living on a virtual guest rather than the local LRMD instance living on the host.
Pacemaker's CRMD component must now be capable of routing LRMD commands to both the local LRMD and pacemaker_remote instances residing on virtual guests.
Defining Linux Container Use-cases:
Part of this project is to identify interesting use-cases for Linux Containers in an HA environment. This project aims at exploring use of libvirt-lxc in HA environments to dynamically define and launch Linux Containers throughout the cluster as well as manage the resources within those containers.
PCS management support for Container Resources:
The pcs management tool for pacemaker needs to support whatever configuration mechanism is settled upon for representing container resources in the CIB.
How To Test
The exact configuration steps necessary to test this feature are not defined. Below are general steps that will be necessary to configure and test the Container Resource on a HA cluster node.
1. Create virtual guest instance (kvm virtual machine or linux container).
2. Configure standalone LRMD daemon to launch in virtual guest on startup.
3. Using pcs on host node, define virtual guest resource and mark it as a container resource.
4. Using pcs on host node, define Dummy resource and mark it as a child resource of the container resource.
5. Once Pacemaker finishes starting the virtual guest resource, the Dummy resource should be launched and monitored within the guest.
Users will be able to define resources as container resources. Once a container resource is defined, that resource becomes a location capable of running resources just as if it was another cluster node.
No new Pacemaker dependencies will be required for this feature. A list of the current dependencies can be found here. 
The part of the project surrounding researching and defining use-cases for Linux Containers in an HA environment runs the risk of being incomplete before Fedora 19. The Linux Container support required to achieve this goal is not well understood and may or may not be possible depending on the capabilities of libvirt-lxc and potentially systemd at the time of release. This outcome will not affect the release this feature however as the KVM use-case is well understood.
If this feature is not complete by development freeze, the functionality necessary to configure this feature will be disabled in the stable configuration scheme. The feature is designed in such a way that it should be able to be disabled without without negatively affecting any existing functionality.
- Upstream documentation specific to this feature will be posted here, Pacemaker Docs, once it is complete.
- Pacemaker now supports the ability to manage resources remotely on non-cluster nodes through the use of the pacemaker_remote service. This feature allows pacemaker to manage both virtual guests and the resources that live within the guests all from the host cluster node without requiring the guest nodes to run the cluster stack. View http://clusterlabs.org/doc/ for more information.