Staging Infrastructure SOP
Fedora has a partially implemented staging environment. This document is intended to be used by people who need to use the staging environment.
Owner: Fedora Infrastructure Team
Contact: #fedora-admin, sysadmin-main
Location: Mostly in PHX2
Servers: app[1-2].stg.phx2.fedoraproject.org proxy1.stg.phx2.fedoraproject.org
Purpose: Provides testing to our admins and community members
Basic workflow and rules
The staging environment replicates as closely as we can the production environment. We don't have full budget to duplicate every single host so we generally pick and choose and do other oddities, for example in staging we combine all the database servers onto one host.
We do use the same username and password in staging and production and regularly merge puppet configs from the master (production) branch to the staging branch. It is also a common practice to dump production databases and pull them into staging. As such no one is given access to staging who doesn't also have it in production.
Pretty much anything goes in staging. If you break something, please fix it. If you're going to do something that breaks other people's work, make sure to let them know first and coordinate accordingly. We try to completely rebuild the staging environment every other release or so.
The addresses in production are identical instaging with the the change s/fedoraproject.org/stg.fedoraproject.org/ So, for example, the staging account system is at https://admin.stg.fedoraproject.org/accounts/. The staging smolts is at http://stg.smolts.org/
In order to make changes to the staging environment you must first checkout the staging branch. This is a pretty standard git branch. From your normal puppet repo (see [Infrastructure/SOP/Puppet]) make sure you are on the right branch by running git branch:
$ git branch * master
Next we're going to check out the staging branch.
$ git checkout -b staging --track origin/staging Branch staging set up to track remote branch staging from origin. Switched to a new branch 'staging'
the -b is the local name in your personal checkout, the --track origin/staging tells git the name of the branch as it is in the master repo. You will see if you run git branch again, git has automatically changed to the staging branch for you:
$ git branch master * staging
To switch back to the master branch just run:
$ git checkout master Switched to branch 'master'
Often times a commit made in staging needs to get pulled into master. You don't want to pull all of staging into master, after all someone else might be working in staging and may not want their changes to go into production yet. To do this cherry-picking is a good alternative. Take the following example where I make a test commit to staging, then cherry-pick it into production:
$ vi README $ git commit -a -m "Made readme more accurate" [staging 52a5150] Made readme more accurate 1 files changed, 1 insertions(+), 2 deletions(-) $ git checkout master Switched to branch 'master' $ git cherry-pick 52a5150 Finished one cherry-pick. [master 8e65e49] Made readme more accurate 1 files changed, 1 insertions(+), 2 deletions(-)
There are 3 git commands run there. First I commit to staging, then I checkout master. Then I cherrypick the git commit into master. From there I can push. Note: I could have pushed the staging commit and tested it in staging which will probably be the most common.
We regularly pull from master into staging to make sure staging doesn't get fully out of sync. In the perfect world the work flow would always go from staging first, into production (master branch) second. Since we cannot fully replicate our production environment, and because we use our staging environment for integration, this isn't possible. So the following command helps keep things in sync:
$ git checkout staging Switched to branch 'staging' $ git merge master
Nodes and the bootstrap
Due to a bootstrap issue in our environment, all staging nodes need to be in both the staging and master branch. To designate a staging node in the staging environment, simply define this variable at the top of the node definition.:
When adding a new node make sure /etc/hosts is all filled out properly for that host and the others (see /etc/hosts below).
Firewalls and /etc/hosts
Because the configs in production are identical to those in staging, we have decided to use some DNS voodoo to get the environment to work correctly. For example, in production a proxy server may contact app[1-7] or bapp1. Since we only have two app servers in staging, we've decided to point app1 prod to app1.stg in staging. Likewise with app2. But app3 points to app1.stg, app4 points to app2.stg, etc.
Also, since the staging environment is on the same physical network as production, we have specific IP based firewall rules in production to deny connections from any staging hosts with a few exceptions (like access to the production yum repos).
What does all of this mean? Well, when creating a new service. Make sure you just use "app1" when referring to app1 or just "db2" when referring to db2. /etc/hosts should take care of the rest. If you are adding a new host. Make sure to add it to every other staring host's node definition. See other staging nodes for examples.