Staging Servers Infrastructure SOP

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(SOP for resyncing the staging fas instance)
Line 36: Line 36:
  
 
{{admon/note| On staging, for files not in modules (ie stuff in configs/*), puppet will use what's in the master branch for those files (TODO: confirm this statement more completely).}}
 
{{admon/note| On staging, for files not in modules (ie stuff in configs/*), puppet will use what's in the master branch for those files (TODO: confirm this statement more completely).}}
 +
 +
== Syncing FAS ==
 +
Sometimes you want to resync the staging fas server with what's on production.  To do that, dump what's in the production db and then import it into the staging db.  Note that resyncing the information will remove any of the information that has been added to the staging fas servers.  So it's good to mention that you're doing this on the infra list or to people who you know are working on the staging fas servers so they can either save their changes or ask you to hold off for a while.
 +
 +
<pre>
 +
On db02:
 +
 +
$ ssh db02
 +
$ sudo -u postgres pg_dump -C fas2 |xz -c fas2.dump.xz
 +
$ scp fas2.dump.xz db01.stg:
 +
 +
 +
On fas01.stg (postgres won't drop the database if something is accessing it)
 +
(ATM, fas in staging is not load balanced so we only have to do this on one server)
 +
 +
$ sudo /etc/init.d/httpd stop
 +
 +
 +
On db01.stg:
 +
 +
$ sudo -u postgres psql 'drop database fas2;'
 +
$ xzcat fas2.dump.xz | sudo -u postgres psql
 +
 +
 +
On fas01.stg:
 +
 +
$ sudo /etc/init.d/httpd start
 +
</pre>
 +
  
 
== Troubleshooting and Resolution ==
 
== Troubleshooting and Resolution ==
  
 
[[Category:Infrastructure SOPs]]
 
[[Category:Infrastructure SOPs]]

Revision as of 14:52, 7 April 2011

Infrastructure InfrastructureTeamN1.png
Shortcut:
ISOP:STAGING

We have the ability to create a staging environment in Infrastructure. This attempts to duplicate a small section of our infrastructure for testing purposes. If there's a large scale change that you need to perform, it's a good idea to push the changes to the staging environment first and check that everything works properly and that you have puppetized all the necessary changes before pushing the changes live.


Contents

Contact Information

Owner: Fedora Infrastructure Team

Contact: #fedora-admin

Persons: mmcgrath

Location: Phoenix

Servers: (proxy1.stg, db1.stg, app1.stg, app2.stg fas1.stg)

Purpose: Staging environment to test changes to apps.

Using

Note.png
When you're on a stg server, you won't have much access to non-stg servers. For example, if you're on proxy1.stg and you ping app1... there's a hosts entry to direct it to app1.stg so it should behave exactly as production does.


Puppet

Staging servers are managed through puppet so that you can test config changes in puppet before pushing live. You need to use a different branch to push to the staging servers. Basic steps:

git clone /git/puppet/
git checkout --track -b staging origin/staging # This only needs to be run once on the initial clone
# make changes, git commits
git push
git checkout master # switch back to the master branch
git checkout staging # switch back to the staging branch
Note.png
On staging, for files not in modules (ie stuff in configs/*), puppet will use what's in the master branch for those files (TODO: confirm this statement more completely).

Syncing FAS

Sometimes you want to resync the staging fas server with what's on production. To do that, dump what's in the production db and then import it into the staging db. Note that resyncing the information will remove any of the information that has been added to the staging fas servers. So it's good to mention that you're doing this on the infra list or to people who you know are working on the staging fas servers so they can either save their changes or ask you to hold off for a while.

On db02:

$ ssh db02
$ sudo -u postgres pg_dump -C fas2 |xz -c fas2.dump.xz
$ scp fas2.dump.xz db01.stg:


On fas01.stg (postgres won't drop the database if something is accessing it)
(ATM, fas in staging is not load balanced so we only have to do this on one server)

$ sudo /etc/init.d/httpd stop


On db01.stg:

$ sudo -u postgres psql 'drop database fas2;'
$ xzcat fas2.dump.xz | sudo -u postgres psql


On fas01.stg:

$ sudo /etc/init.d/httpd start


Troubleshooting and Resolution