Cloud/Cloud PRD

CURRENTLY EDITING IN ETHERPAD. SEE #fedora-cloud on Freenode to coordinate.

Fedora Cloud Product Requirements Document.

Document Purpose and Overview

This is the wikipedia Product Requirements Document for the Fedora Cloud SIG. It:

• Provides a high-level market of the cloud computing market as it pertains to the Fedora Cloud SIG; this includes overviews of things which may not be within our actual scope/ability to accomplish at the current time.
• Provides deeper understanding of the types of users who could use Fedora for their cloud computing needs. This includes describing their main day-to-day tasks, common problems, and so on. The perspective here is not necessarily limited to system administrators, or developers, but a combination of many types of users and roles.
• Ties common issues and needs of potential users and consumers of the Fedora Cloud product to high-level product needs, from a "functional" standpoint
• Provides solutions, in the form of "changes" or "features" which will provide the functionality described as needs for the potential users.

Release/Product Overview

Fedora Cloud provides a customizable base image and tools for developing scale out applications on public and private clouds, as well as two to four images pre-configured for what we expect to be the most popular scenarios/use cases for using Fedora for cloud computing. Public and private cloud adoption is taking off, and the requirements for an image OS differ significantly from the requirements for a desktop or server OS. In these environments, much or all of the instance lifecycle — from the creation of the image and addition of software or configuration specific to the instance, to the teardown of the image — will be automated. Systems are designed to scale out via many identical nodes rather than scale up with carefully-tended individual servers. Individual uptime (mean time between failure) is not as important as the ability to get a new instance running quickly (mean time to recovery). With that in mind, we're tailoring a release specifically for cloud environments.

Market Opportunity

Public and private cloud adoption is happening rapidly, but the market is not yet mature and is relatively ripe for disruption even though some favorites have emerged as early leaders. In the next two to three years, we expect to see a great deal of growth in adoption and still see a number of emerging players where no clear favorites have emerged (for instance, Google Compute Engine). Additionally, some platforms (Amazon Web Services) have matured to a point where a large number of companies are relying upon the technology for their full infrastructure. While this is not a widespread practice, AWS is seeing a great deal of adoption and will likely start eating into "traditional" workloads that currently live behind the firewall. In short, there's an enormous opportunity for Fedora to become an instance-OS of choice if the project moves quickly, develops or adopts the right technologies, and succeeds in educating the market about its existence. A failure on any of those three points means that the Fedora Cloud product will have little chance in taking a significant portion of the new market or taking any of the existing market.

Major Release Themes

Cloud computing in general is the transition of computing power from individual hand-tended resources to a ubiquitous utility. Fedora fits in at several levels, from the infrastructure service software we include (like OpenStack and Eucalyptus) to end-user tools. The Fedora Cloud image fits into middle of this, providing a guest OS image to run on Infrastructure as a Service systems, on which platform and application services can be deployed. We targets use cases which fit the "cattle" side of the "pets vs. cattle" metaphor for computing.

Secondary Objectives

Aside from adoption and development of applications on top of the Fedora Cloud images, we have a few secondary goals that will be helped by wider adoption:

• More testing of Fedora images with additional bug reports.
• Better feedback about how the product should improve. This is separate from "bug reports" in that we hope to engage the audience and receive detailed feedback about use cases, desired features, developing trends in cloud management, etc.
• Patches and contributions that will help improve the product, and Fedora in general. As we are increasingly successful, more users will take an interest in helping to develop our product.

Target Market / Audience

Developers creating scale out applications on top of public and private clouds, and organizations and users running those applications. FIXME (this could use another sentence? or if you think it is both beautiful and sufficient, feel free to just delete this FIXME)

Delivery Mechanisms

Cloud images images must be easy to consume as part of a pulbic or private cloud. Because we target these environments, we won't be worrying about physical media at all. The cloud images also won't be "installed" in the same manner that users are accustomed to with desktop or server images. The cloud image will simply boot in its target environment ready to run, or for further customization and configuration via a boostrapping service.

Where to obtain

Users will be able to obtain the images for public clouds via download or via the usual marketplace for those images. For instance, we publish Amazon AMIs on Amazon directly. Users are able to launch new instances with Fedora without having to obtain the images directly from the Fedora Project and then upload to Amazon. Users will be able to download appropriate images for Apache CloudStack, Eucalyptus, OpenStack, OpenNebula, and other IaaS platforms.

Delivery Format

Images will be delivered as AMIs on Amazon EC2, and as downloadable images in qcow2 and raw.xz formats. We may add other public cloud images and other downloadable formats to meet demand or anticipated need.

Updates

Fedora Cloud images can be updated using yum as normal. We also intend to produce periodic respun images with security updates, once the infrastructure is in place to support that.

Image Creation Toolkit

We will also maintain a set of tools that can be used to generate, modify, and configure Fedora instances for use with public and private clouds. Initial efforts will focus on creating official images in Fedora's build system. This effort is in parallel to that and does not block the main release. The Fedora Cloud SIG is also interested in a curated library of image templates.... FIXME BLAH BLAH BLAH

Measuring Success

Currently, Fedora is not a widely used option for instances on public and private clouds. We know there's some usage, but it's not one of the top three or four OSes on Amazon or (likely) for private clouds. Success looks like:

• Increase in adoption.
• Third party support / targeting of Fedora Cloud as a platform.
• Increased contribution and participation in the Fedora Cloud WG and Fedora Project in general.

User Profiles, Goals, and Primary Use Cases

User Profiles

FIXME: -- integrate this list, make pretty-sounding paragraph to introduce it Three Cloud User Roles (based on “Description and Application of Core Cloud User Roles” ACM CHIMIT 2011, December 4 2011) that describe the tasks of the people who interact with any cloud-based Information Technology system:

1. Cloud Service Creator: Develop the technical and business aspects of a (simple or high-level) cloud service
2. Cloud Service Provider: Provide all types of services (SPI, etc.) to a Service Consumer
3. Cloud Service Consumer: Consume all types of services (SPI) offered by a Service Provider

We will use a set of personas to describe our target users and their respective needs. This document will list the personas in their simplified forms, with detailed explanations of each one available on their respective wiki pages. https://fedoraproject.org/wiki/Cloud_Personas

1. AWS enthusiast and Early Adopter
   • Writes and maintains a number of AWS-deployed applications including staging and production.
   • Works for a small start-up, primarily on his own. Automates as much as possible.
2. Rails DevOps Team Member
   • Uses the cloud to do Ruby on Rails development utilizing virtual machines.
   • Works as part of a DevOps team responsible for all aspects of a set of applications.
   • Development, QA, and Production environments need to be identical.
3. Web Developer
   • Concentrates on app development, not architecture, deployment, or operations.
   • FIXME
4. Large-Environment Sysadmin
   • Interested in deploying self-service PaaS to lighten operational load
   • FIXME
5. Data Scientist
   • Works with large data sets and intends to
   • FIXME
6. HPC Scientist
   • Moving from traditional batch and grid technology to the cloud
   • FIXME

Primary Use Cases

FIXME: does this list cover all of our personas?

Web Application Deployment in a Public Cloud

Modern web applications are deployed as a collection of interconnected services, including parts like web servers, application servers, databases, and caching layers. Fault tolerance is handled at the overall orchestration level rather than by individual instances Fedora Cloud can be the base of each of these parts, providing recent libraries, server software, and language toolchains. Each system will be managed using the public cloud's own management tools plus a configuration management system like Puppet, Chef, Salt, or Ansible. Code may be developed against different language stacks; if these are readily available to users, that's a win. In the real world, libraries and other code we don't include may also be required and we should make that as painless as possible. FIXME: we mention Rails in our personas. Let's call that out specifically, as well as a few other major language stacks we know are priorities.,

Web Application Deployment in Private Cloud

As above, but in a locally-deployed and managed private cloud system.

Web Application Deployment in Hybrid Cloud

As above, but rather than a single cloud provider, the application seamlessly takes advantage of resources in both a private and public cloud.

Hybrid Cloud as Staging Area

Application development and testing happens in-house on a private cloud, while production goes to a public cloud. FIXME: MORE DETAILS

Big Data / Number-Crunching

Deploy scale-out application for data processing to public, private, or hybrid cloud. FIXME: more? Deetails????

OpenShift Origin System

FIXME: make this tech-independent -- basis for PAAS. (OpenShift can be mentioned, but these should be problem-focused.) OpenShift Origin is an open source platform-as-a-service system already included in Fedora. A Fedora Cloud image focused on OpenShift would make it easy for users to run their own PaaS.

Simple Deployment of code to from Development to Production

As a developer, I want to ensure that my code is easily deployable from my development environment to a QA environment and finally a production environment, without encountering compatibility issues or surprises from differences in the environment. A solution where an identical customized image or container is used in all three environments is ideal.

Development as Part of a Team

As a member of a development team, I would like to develop in an environment where code can go through unit or functional testing, and be approved or accepted by other members of the team. Example: A feature described as “Continuous Integration platform” may be listed in the Features section, and the various tools available to implement would be enumerated and described in the “non-functional requirements” section, and cross-coordinated with the server working group. FIXME/note I (mattdm) don't think we're ready for this one -- it's more involved than the other things we are tackling. maybe after the first release?

Target IaaS environments

The Fedora Cloud product can be used as a guest/VM/image under many IaaS services and providers. For projects that are not currently packaged within Fedora/EPEL, we may need to locate a kind person to ensure testing. These include: Open source IaaS systems:

• OpenStack
• Eucalyptus
• Apache Cloudstack
• OpenNebula
• oVirt (FIXME: should this be in our focus? it's more traditional enterprise virt than iaas)

Public clouds:

• Amazon EC2
• Google Compute Engine
• HP Cloud
• Rackspace
• Digital Ocean
• Linode

Features

FIXME: this is documentation for us as PRD writers. Remove once the feature section is in better shape. :) Features here address the primary and secondary use cases, product or secondary objectives, market opportunities from above. Features should provide functional requirements (“what it should do”) preferably in a narrative fashion - more of a story / solution description, rather than “package XYZ” - the features (the ways to meet a user's objectives?) each likely consist of more than one package/enhancement, and those packages should be detailed in the “Detailed requirements” section of this document, and each of those detailed requirements should refer back to which feature it supports.

Feature #0

Feature description should be described in the line saying “Feature #1/2/etc.”. Describe the feature in more detail, specifically addressing how it addresses user scenarios, primary or secondary use cases / objectives of the product. Use a table to indicate the following items: Priority (Must, Should, NTH) Citation of use cases addressed As work continues and specific detailed requirements are developed, reference the detailed requirements within this document helping to fulfill this feature. This helps to ensure awareness around “do we still have a feature if some of the detailed requirements are not fulfilled, and thus are not able to address the specific use case needs / user objectives.”

Feature: Ready to run in Public and Private Clouds

The Fedora cloud image is ready to go out of the box in the public and private cloud environments we target. It includes cloud-init, the defacto standard for boot-time configuration for cloud instances, and the client provisioning tools for OpenStack Heat.

Feature: Ready Access to Fedora Collection of Packages

SUPER FIXME

• normal packages
• language and application stacks to enable whatever you need... blah blah blah FIXME
• different versions of languages through Software Collections
• other technologies as they are developed and integrated

Feature: Docker support

Docker is an easy to use interface for running application containers on Linux. Fedora is uniquely positioned to provide the best platform for Docker, since this container technology is not a security solution, but can be made reasonably secure when wrapped with SELinux. This includes adding libvirt support to the image, which is more heavyweight than many users of the image for other purposes will want, so we will produce a dedicated image specfically tailored for Docker.

Feature: Big Data Tools

We will produce a specialized image with various tools for Big Data processing preloaded and preconfigured where possible. The exact composition of this image will be determined in collaboration with Fedora's Big Data SIG.

Feature: Quick OpenShift Deployment

FIXME: ...

Feature: Cloud -> Server

The Fedora Server product targets more traditional server roles, where systems have a more unique identity. The Fedora Cloud image supports this by providing a path to go from our base to a Fedora Server role, in effect taming one of your cloud computing "cattle" and making it into a "pet" traditional server — but running in a cloud environment.

Future Features

The following areas are unlilel;y to be completely addressed in the upcoming iteration of Fedora Cloud, but we intend to work on them in the future.

Image Generation Tools

FIXME

Image or Image Template Library

FIXME

Higher-Level Cloud Tooling

FIXME

1. Orchestration
2. Performance / Scalability / Failover
3. Logging / Auditing
4. Monitoring / Notification
5. Database requirements

Requirements

FIXME TODO: change these things into actual requirements in the lovely format shown below.

• ✓ Support for smallification: a) kernel (reqs for kernel team), b) lang & docs (reqs for packaging tools team), c) cloud-init refactoring
• ✓ Docker security ( = strong selling point): Libvirt/SELinux integration for Docker (planned but not done)
• big data spin - any needs?
• Infrastructure for automatic production and upload of cloud images for updates
• easy software stacks -- install your language of choice, preferably your version of things of choice
• integration with Fedora Server roles
• tools for user creation of images -- imagefactory?
  • repository for these images, or at least their definitions?
  • dockerfiles? same or different?
  • DOCUMENTATION -- for image creation, cloud-init, what to use when, etc.
• web site design for selecting version to launch or download -- specs to web team

Requirement #1 (Short Description of Requirement)

Feature(s) Addressed

Refer to which previously described Features, Use Cases this requirement helps to fulfill.

Priority

Must, should, NTH

Effort required

High, Medium, Low

Major Dependencies

Any major dependencies, including things that may require any cross-working-group coordination, should be called out here. Any process changes required within Fedora should be documented here as well.

Testing

Level of testing required; is it a blocker to release? Is the testing automate-able?

Other Documentation

• Existing BZ:
• Upstream webpage / wiki page / github page(s):

Requirement #N: Reducing Image Footprint

FIXME Short description

Priority

Nice to Have. The current cloud image is reasonably sized when compared to our competitors. However, smaller footprint has several advantages. Fewer packages means fewer updates and a smaller target for security problems. It makes it faster to download and deploy images. And, reducing things our users don't really need gives more room to include things they do.

Specific Areas

This requirement has several areas where effort will yield meaningful results. Each has its own level of effort, stakeholders, and dependencies.

Cloud-Focused Base Kernel

• FIXME

= Internationalization / Localization

• FIXME NTH: Cloud Product base images should only includes en.us_US locale since it is meant to be used as an deployment target to save space. Other locales should be available through repositories.

The idea is to rely on langpacks rather than RPM wizardry.

Included Documentation

• FIXME... same problem

= Refactoring Cloud-Init=

• FIXME Dependency chain craziness, modules not necessarily right, etc.

Requirement #N: Improved Docker Integration

FIXME -- particularly of interest, selinux support. also docker infrastructure, image building, dockerfiles, etc. == Requirement #N: Automatic Production and Upload of Images

FIXME: describe. qa also here? (taskotron?)

Documentation

Open Source Projects documentation

Ensuring that Fedora is well-represented, up-to-date in other open source project documentation...

Release Criteria

FIXME -- should we punt on this for now?

Technical requirements

Support Requirements

Architectures Supported

• x86
• x86_64
• at some point: ARM (which variants ?)

Supported (platforms?)

Virtualization types? Container types?

Security

FIXME -- also punt?

About this Document

This PRD (Product Requirements Document) is an evolving document, created by the Fedora Cloud SIG Working Group as part of the process for designing the Fedora Cloud product. The framework for the PRD itself is currently in a draft state. This document will evolve over time as the purpose of the SIG continues to be determined as the working group process gets under way and initial products start to get produced.

Authors

Contributors to this document include:

• Robyn Bergeron
• Matthew Miller
• Sam Kottler
• Haïkel Guémar

Some aspects of Fedora Cloud personas are based on OpenStack personas (licensed under CC-By 3.0).

Reviewers & Contributors

The following people have contributed to the development of this document, through feedback on IRC, mailing lists, and other points of contact.

• Matthew Miller

Community Information

The Fedora Cloud SIG is one of many teams within the Fedora Project. The Cloud SIG mailing list is located here. Minutes and logs from IRC meetings related to the development of this document should be listed here as the document evolves.

Approval History

Over time, it is expected that this document will undergo various rounds of review, approval, and editing; in the future, it may be rewritten for different releases of Fedora. While one can review the history of a wiki document (by clicking the "history" tab), it is useful to provide explicit indicators of any major format changes, approvals, or indications of it being in a “final” state, in a list that can allow someone to quickly see that all of the prescribed layers of approval have occured.

• January 8, 2014: Collaborative hackfest day.
• October 28, 2013: Initial Draft of template.
• FutureDate: Approval by SomeGroup; link to any pertinent mail announcement and/or meeting minutes

Tracking of Progress

This document contains numerous descriptions of use cases, descriptions of feature sets to address the use cases, and the requirements to enable those features. Numerous Fedora self-contained and systemwide changess (in addition to updates to individual packages) may combine to address those use cases and feature sets. Thus, as a single release, or even series of releases, undergoes development, it is useful to easily track how an entire use case or feature set may be progressing. While Fedora uses the Changes Process to track changes in the distribution, those changes are typically described as details of changes to a specific package, or the introduction of a specific package, rather than as a piece of a larger feature set. This document could possibly be used to do any or a number of the following things:

• Provide a secondary location where changes are tracked (which seems like a lot of overhead to me)
• Provide a location where overall Feature Progress is tracked, via periodic cross-checking against Change pages; this could be either in a standalone section, or simply attached to each Feature description.
• Scope out how features are expected to progress over a number of releases.
• None of these things.

FIXME -- add this When we more fully determine how to most efficiently track progress, the pointer to where that tracking is done, and/or the description of or process by which we do the tracking is formalized, should be documented in this section in lieu of what is currently written here.

Document Conventions

Definitions and Acronyms

• AWS: Amazon Web Services
• Amazon EC2: Amazon Elastic Compute Cloud, a popular public IaaS
• IaaS: Infrastructure as a Service
• PaaS: Platform as a Service
• SaaS: Software as a Service
• PRD: Product Requirements Document
• EPEL: Extra Packages for Enterprise Linux
• CI: Continuous Integration
• CD: Continuous Delivery or Continuous Deployment
• SCL: Software Collections

  teams in charge of some aspects of Fedora Project

• NTH: Nice-to-have
• BZ: Bugzilla
• GUI: Graphical User Interface
• CLI: Command Line Interface
• API: Application Programming Interface