From Fedora Project Wiki
 
(41 intermediate revisions by 3 users not shown)
Line 1: Line 1:
= First-Class Cloud Images =
= First-Class Cloud Images =
<span style="font-size:200%">
This feature is complete. You can now get Fedora Cloud Images from [http://cloud.fedoraproject.org/ http://cloud.fedoraproject.org/].</span>


== Summary ==
== Summary ==
Cloud images for EC2 and for generic use (download and use in any cloud environment, like OpenStack, CloudStack, or Eucalyptus) will be produced as part of the Alpha, Beta, and Final compose process, and distributed on the mirror network. We'll also have automatic nightly builds of Rawhide and of the  development version.
This feature expands Fedora's current cloud image deliverables beyond just EC2, and overhauls how they are produced. The goal is to produce cloud images for EC2 and other cloud deployments for the Alpha, Beta, and Final compose process and distribute them on the mirror network. There will also be nightly or weekly image builds for Rawhide to assist with early development. All images should be constructed using a newer generation of tools.


== Owner ==
== Owner ==
* Name: [[User:mattdm| Matthew Miller]]
* Name: [[User:mattdm| Matthew Miller]]
* Email: mattdm at fedoraproject dot org
* Email: mattdm at fedoraproject dot org


== Current status ==
== Current status ==
* Targeted release: [[Releases/19 | Fedora 19 ]]  
* Targeted release: [[Releases/19 | Fedora 19 ]]  
* Last updated: January 19, 2013
* Last updated: 2013-05-14
* Percentage of completion: 0%
* Percentage of completion: (see text)
 
We weren't able to complete the idea of using Oz in Koji, so we are activating that aspect of the contingency plan. Some of the automation is also not yet in place. However, the existing build tech has been patched so that images can be built for both EC2 and for download for use in OpenStack and etc. That's the core part of the feature and '''this will be as planned, from F19 Beta on'''  - notably, the release notes section below is still accurate.
 
We do still need to move to an anaconda-based install system, and the remaining aspects of this feature will be re-proposed for F20.


== Detailed Description ==
== Detailed Description ==
* Koji updated to support image creation under visualization using Factory/Oz
* New images that can be used in other cloud deployments (such as [http://openstack.org OpenStack], [http://incubator.apache.org/cloudstack/ CloudStack], or [http://www.eucalyptus.com/eucalyptus-cloud Eucalyptus]) will be produced. They will be in a qcow2 format and lack the EC2-specific customization. Images for this feature would ideally work for all cloud deployments and there will be i686 and x86_64 versions of both image types. In total and "image drop" will have 4 images: 2 arches for 2 different types (EC2, not-EC2).
* Scratch images built weekly
* An image drop will be produced for Alpha, Beta, and Final composes for Fedora 19 and forward.
* "Blessed" built for Alpha, Beta, and Final
* Scratch build image drops will be produced on a weekly basis for Fedora 19.
* Scratch build image drops will be produced on a weekly basis for Rawhide as well to enable early testing.
* The Fedora Koji instance needs to be updated to a future release that will integrate with [http://imgfac.org ImageFactory and Oz] from the [http://www.aeolusproject.org/ Aeolus Project]. This future release is not implemented yet.
* The EC2 images will be automatically uploaded and registered in EC2. The Final AMIs for Fedora 19 will be available in the Amazon marketplace.


== Benefit to Fedora ==
== Benefit to Fedora ==
* Cloud images more easily available to users
* Cloud images more easily available to users
* Cloud images available for better testing
* Cloud images available for better testing
* Constant building of images provides better platform for testing in general
* Continuous building of images provides new opportunities for early platform testing
* appliance tools, which is the workhorse for image building today, does not have an upstream. [http://imgfac.org ImageFactory] does.
* appliance-tools uses chroots which suffer from build-time complications like kernel mismatches. Moving off of this tool will unburden Rel-Eng with that work.


== Scope ==
== Scope ==
Updates to Koji are a fairly major infrastructure change. We'll need bare-metal builders (or nested virt). ImageFactory/Oz will grow the ability to take kickstarts in addition to xml templates. Creating LiveCDs with the same system will also require some changes to ImageFactory/Oz.


Release engineering will produce images weekly (in an automated way).
==== ImageFactory/Oz Integration with Koji ====
Creating LiveCDs with the same system will also require some changes to ImageFactory/Oz. These will use existing technology in [http://lists.fedoraproject.org/pipermail/devel/2011-December/160657.html livemedia-creator] (not to be confused with livecd-creator). ImageFactory/Oz will need to be installed on the build hosts, and kojid will make use of them when it takes an image building task.
 
ImageFactory/Oz must be capable of consuming raw kickstarts for this feature. Use of the XML templates will be reviewed at a later date.
 
==== Build System Update Deployment ====
This feature requires a significant change to Koji that will need to be deployed to the production build system. ImageFactory/Oz builds the images inside a small VM, and because a nested virt scenario is not possible on RHEL 6 (which is what the builders are) it will require bare metal builders to be available.
 
There are 2 bare-metal builders available today to accommodate this requirement. (thank you Dennis Gilmore)
 
==== Process and Infrastructure Updates ====
Release Engineering will produce image drops on a weekly basis and for milestone updates. These will need to be easily discoverable so that announcements and communication about their release is easily consumed. (from the Cloud SIG web page, for example). Procedures for producing, testing, and blessing the images should be documented and communicated as well.
 
Milestone image drops will be released using the current Fedora mirroring system, alongside the install images.
 
==== Updated Web Information ====


Will also need procedure for producing, testing, and blessing the official images for test and final releases.
Current [http://fedoraproject.org/en/get-fedora-options#cloud Get Fedora in the Cloud] web sites are focused around EC2. This needs to be redesigned to present multiple options. Additionally, the EC2 page should have a clickable launch button.


== How To Test ==
== Milestones ==
<!-- This does not need to be a full-fledged document.  Describe the dimensions of tests that this feature is expected to pass when it is done.  If it needs to be tested with different hardware or software configurations, indicate them.  The more specific you can be, the better the community testing can be.


Remember that you are writing this how to for interested testers to use to check out your feature - documenting what you do for testing is OK, but it's much better to document what *I* can do to test your feature.
* <strike>Last chance to get Imagefactory/Oz/livemedia-creator working in a chroot</strike> (Determined to be not workable with reasonable amount of effort.): February 4th, 2013.
* Prototype implementation of Koji-Imagefactory/Oz: February 28th, 2013
* Code landed in upstream Koji project: March 19th, 2013 (Allowing one week to update builders.)
* Fedora Koji builders updated: March 26th, 2013 (One week before Alpha freeze.)
* Scripting for automatic (Rawhide/F19) weeklies: April 19th, 2013
* Weekly image drops made available on http://alt.fedoraproject.org/pub/alt/cloud/: April 26rd, 2013 (First weekend after alpha release)
* Image drops for Alpha, Beta, and Final milestones will be on the mirrors on the same date as those milestones.


A good "how to test" should answer these four questions:
== How To Test ==
Since images are composed of packages it would be redundant to test all package functionality in each image in each cloud environment. It should be sufficient to verify that the image boots and is capable of getting yum updates if the cloud environment is configured to provide them or the image has network access to the internet.


0. What special hardware / data / etc. is needed (if any)?
* Do the EC2-specific images exist in EC2?
1. How do I prepare my system to test this feature? What packages
** Are they bootable?
need to be installed, config files edited, etc.?
** Can one log in with the appropriate user account (probably ec2-user) with a provided ssh key?
2. What specific actions do I perform to check that the feature is
** Does a yum update successfully retrieve updates?
working like it's supposed to?
* Are the non-EC2 images available?
3. What are the expected results of those actions?
** expected formats: qcow2, raw.tar.xz
-->
** Do the downloadable images boot in OpenStack?
** Do the downloadable images boot in Eucalyptus?
** Do they appear similar to one installed by Anaconda?
** Are they On the mirrors for Alpha, Beta, and Final?


== User Experience ==
== User Experience ==
<!-- If this feature is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
Milestone ("Official") cloud images must be downloadable from the mirror system. They should also be well announced and discoverable, perhaps on a wiki somewhere. The EC2 images must be registered as AMIs and browsable in the AWS console in all regions. It is desirable to see them in the Quickstart guide too, or the Amazon Marketplace at no additional charge beyond the usual infrastructure costs.
 
Weekly Rawhide images should follow the same criteria above minus the Quickstart and Marketplace items.


== Dependencies ==
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this feature depends?  In other words, completion of another feature owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel feature)? -->
The ImageFactory/Oz changes are pending upstream approval.
 
The Koji integration with ImageFactory/Oz is pending upstream approval.


== Contingency Plan ==
== Contingency Plan ==
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "None necessary, revert to previous release behaviour."  Or it might not. If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy. -->
 
Since this feature involves a few moving parts, they each have their own contingency plan.
 
==== Koji Integration ====
If the Koji integration work is not completed before the Fedora 19 Alpha deadline, we can generate EC2 images for the Alpha in the old way and skip having ''official'' non-EC2 images. However, those images will still be built using ImageFactory and Oz out of Koji and released through the Cloud SIG.
 
If we miss both that ''and'' updating Koji for the Beta deadline, we'll produce images in the old way (with no official non-EC2 images for beta or final) and revisit for F20.
 
As a middle ground if the integration work is unexpectedly delayed, consider updating Fedora Koji to the 1.7.1 release instead, which tracks images using the same data model as RPMs. This will at least improve the manageability if the images produced and enable some level of automation to track the latest.
 
Livemedia-creator could be used instead of Oz, but since the integration work is about the same, it would be pointless to begin that task late in the Fedora 19 release cycle.
 
==== Continuous Image Building ====
Building images requires a fair amount of disk space. If it proves to be too aggressive, we could throttle their creation to a bi-weekly task, or make the lifetime of the scratch images shorter.


== Documentation ==
== Documentation ==
<!-- Is there upstream documentation on this feature, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
 
*
See [[Features/FirstClassCloudImages/Whiteboard]]
 
Possibly a small readme file should go alongside the images. Primary documentation on [[Cloud SIG]] web page.


== Release Notes ==
== Release Notes ==
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are here: http://docs.fedoraproject.org/release-notes/ -->
 
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this feature, indicate them here.  You can also link to upstream documentation if it satisfies this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release. -->
Ready-to-run cloud images are provided as part of this Fedora release. These are available in Amazon EC2 and for direct download from http://cloud.fedoraproject.org/. The downloadable images are available in compressed raw image format and qcow2 for immediate use with OpenStack. The images are configured with cloud-init, and so will take advantage of ec2-compatible metadata services for provisioning ssh keys.
*


== Comments and Discussion ==
== Comments and Discussion ==
* See [[Talk:Features/Your_Feature_Name]] <!-- This adds a link to the "discussion" tab associated with your page.  This provides the ability to have ongoing comments or conversation without bogging down the main feature page -->
* See [[Talk:Features/FirstClassCloudImages]]
 


[[Category:FeaturePageIncomplete]]
[[Category:FeatureAcceptedF19]]
<!-- When your feature page is completed and ready for review -->
<!-- remove Category:FeaturePageIncomplete and change it to Category:FeatureReadyForWrangler -->
<!-- After review, the feature wrangler will move your page to Category:FeatureReadyForFesco... if it still needs more work it will move back to Category:FeaturePageIncomplete-->
<!-- A pretty picture of the page category usage is at: https://fedoraproject.org/wiki/Features/Policy/Process -->

Latest revision as of 21:48, 3 July 2013

First-Class Cloud Images

This feature is complete. You can now get Fedora Cloud Images from http://cloud.fedoraproject.org/.

Summary

This feature expands Fedora's current cloud image deliverables beyond just EC2, and overhauls how they are produced. The goal is to produce cloud images for EC2 and other cloud deployments for the Alpha, Beta, and Final compose process and distribute them on the mirror network. There will also be nightly or weekly image builds for Rawhide to assist with early development. All images should be constructed using a newer generation of tools.

Owner

Current status

  • Targeted release: Fedora 19
  • Last updated: 2013-05-14
  • Percentage of completion: (see text)

We weren't able to complete the idea of using Oz in Koji, so we are activating that aspect of the contingency plan. Some of the automation is also not yet in place. However, the existing build tech has been patched so that images can be built for both EC2 and for download for use in OpenStack and etc. That's the core part of the feature and this will be as planned, from F19 Beta on - notably, the release notes section below is still accurate.

We do still need to move to an anaconda-based install system, and the remaining aspects of this feature will be re-proposed for F20.

Detailed Description

  • New images that can be used in other cloud deployments (such as OpenStack, CloudStack, or Eucalyptus) will be produced. They will be in a qcow2 format and lack the EC2-specific customization. Images for this feature would ideally work for all cloud deployments and there will be i686 and x86_64 versions of both image types. In total and "image drop" will have 4 images: 2 arches for 2 different types (EC2, not-EC2).
  • An image drop will be produced for Alpha, Beta, and Final composes for Fedora 19 and forward.
  • Scratch build image drops will be produced on a weekly basis for Fedora 19.
  • Scratch build image drops will be produced on a weekly basis for Rawhide as well to enable early testing.
  • The Fedora Koji instance needs to be updated to a future release that will integrate with ImageFactory and Oz from the Aeolus Project. This future release is not implemented yet.
  • The EC2 images will be automatically uploaded and registered in EC2. The Final AMIs for Fedora 19 will be available in the Amazon marketplace.

Benefit to Fedora

  • Cloud images more easily available to users
  • Cloud images available for better testing
  • Continuous building of images provides new opportunities for early platform testing
  • appliance tools, which is the workhorse for image building today, does not have an upstream. ImageFactory does.
  • appliance-tools uses chroots which suffer from build-time complications like kernel mismatches. Moving off of this tool will unburden Rel-Eng with that work.

Scope

ImageFactory/Oz Integration with Koji

Creating LiveCDs with the same system will also require some changes to ImageFactory/Oz. These will use existing technology in livemedia-creator (not to be confused with livecd-creator). ImageFactory/Oz will need to be installed on the build hosts, and kojid will make use of them when it takes an image building task.

ImageFactory/Oz must be capable of consuming raw kickstarts for this feature. Use of the XML templates will be reviewed at a later date.

Build System Update Deployment

This feature requires a significant change to Koji that will need to be deployed to the production build system. ImageFactory/Oz builds the images inside a small VM, and because a nested virt scenario is not possible on RHEL 6 (which is what the builders are) it will require bare metal builders to be available.

There are 2 bare-metal builders available today to accommodate this requirement. (thank you Dennis Gilmore)

Process and Infrastructure Updates

Release Engineering will produce image drops on a weekly basis and for milestone updates. These will need to be easily discoverable so that announcements and communication about their release is easily consumed. (from the Cloud SIG web page, for example). Procedures for producing, testing, and blessing the images should be documented and communicated as well.

Milestone image drops will be released using the current Fedora mirroring system, alongside the install images.

Updated Web Information

Current Get Fedora in the Cloud web sites are focused around EC2. This needs to be redesigned to present multiple options. Additionally, the EC2 page should have a clickable launch button.

Milestones

  • Last chance to get Imagefactory/Oz/livemedia-creator working in a chroot (Determined to be not workable with reasonable amount of effort.): February 4th, 2013.
  • Prototype implementation of Koji-Imagefactory/Oz: February 28th, 2013
  • Code landed in upstream Koji project: March 19th, 2013 (Allowing one week to update builders.)
  • Fedora Koji builders updated: March 26th, 2013 (One week before Alpha freeze.)
  • Scripting for automatic (Rawhide/F19) weeklies: April 19th, 2013
  • Weekly image drops made available on http://alt.fedoraproject.org/pub/alt/cloud/: April 26rd, 2013 (First weekend after alpha release)
  • Image drops for Alpha, Beta, and Final milestones will be on the mirrors on the same date as those milestones.

How To Test

Since images are composed of packages it would be redundant to test all package functionality in each image in each cloud environment. It should be sufficient to verify that the image boots and is capable of getting yum updates if the cloud environment is configured to provide them or the image has network access to the internet.

  • Do the EC2-specific images exist in EC2?
    • Are they bootable?
    • Can one log in with the appropriate user account (probably ec2-user) with a provided ssh key?
    • Does a yum update successfully retrieve updates?
  • Are the non-EC2 images available?
    • expected formats: qcow2, raw.tar.xz
    • Do the downloadable images boot in OpenStack?
    • Do the downloadable images boot in Eucalyptus?
    • Do they appear similar to one installed by Anaconda?
    • Are they On the mirrors for Alpha, Beta, and Final?

User Experience

Milestone ("Official") cloud images must be downloadable from the mirror system. They should also be well announced and discoverable, perhaps on a wiki somewhere. The EC2 images must be registered as AMIs and browsable in the AWS console in all regions. It is desirable to see them in the Quickstart guide too, or the Amazon Marketplace at no additional charge beyond the usual infrastructure costs.

Weekly Rawhide images should follow the same criteria above minus the Quickstart and Marketplace items.

Dependencies

The ImageFactory/Oz changes are pending upstream approval.

The Koji integration with ImageFactory/Oz is pending upstream approval.

Contingency Plan

Since this feature involves a few moving parts, they each have their own contingency plan.

Koji Integration

If the Koji integration work is not completed before the Fedora 19 Alpha deadline, we can generate EC2 images for the Alpha in the old way and skip having official non-EC2 images. However, those images will still be built using ImageFactory and Oz out of Koji and released through the Cloud SIG.

If we miss both that and updating Koji for the Beta deadline, we'll produce images in the old way (with no official non-EC2 images for beta or final) and revisit for F20.

As a middle ground if the integration work is unexpectedly delayed, consider updating Fedora Koji to the 1.7.1 release instead, which tracks images using the same data model as RPMs. This will at least improve the manageability if the images produced and enable some level of automation to track the latest.

Livemedia-creator could be used instead of Oz, but since the integration work is about the same, it would be pointless to begin that task late in the Fedora 19 release cycle.

Continuous Image Building

Building images requires a fair amount of disk space. If it proves to be too aggressive, we could throttle their creation to a bi-weekly task, or make the lifetime of the scratch images shorter.

Documentation

See Features/FirstClassCloudImages/Whiteboard

Possibly a small readme file should go alongside the images. Primary documentation on Cloud SIG web page.

Release Notes

Ready-to-run cloud images are provided as part of this Fedora release. These are available in Amazon EC2 and for direct download from http://cloud.fedoraproject.org/. The downloadable images are available in compressed raw image format and qcow2 for immediate use with OpenStack. The images are configured with cloud-init, and so will take advantage of ec2-compatible metadata services for provisioning ssh keys.

Comments and Discussion