From Fedora Project Wiki

No edit summary
No edit summary
 
(40 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Fedora 19 is when OpenShift Origin first became a feature.
Fedora 19 is when OpenShift Origin first became a feature.
NOTE: (August 8, 2013) This page is getting an update.  It will accommodate F19 cloud images (not just minimal install).  It is also updated with the OpenShift Origin Version 2 documentation.


This page is here to show how to setup OpenShift Origin on Fedora 19 using the packages in Fedora, as opposed to the packages published from upstream.  These steps are written out to be done by hand.  Yes, people can script and/or puppetize these steps.  But these are written out so that people can see, and fine tune them.
This page is here to show how to setup OpenShift Origin on Fedora 19 using the packages in Fedora, as opposed to the packages published from upstream.  These steps are written out to be done by hand.  Yes, people can script and/or puppetize these steps.  But these are written out so that people can see, and fine tune them.


** Starting with Fedora Minimal install
Note: And now they have been written into scripts. https://github.com/tdawson/oo-install-scripts
https://www.openshift.com/wiki/build-your-own
https://www.openshift.com/forums/openshift/fedora-18-openshift-origin-setup-steps-and-testing
=1== Setup System =1==
# ON BROKER
yum -y update
= avoid clock skew
yum -y install ntp
/bin/systemctl enable ntpd.service
/bin/systemctl start  ntpd.service


=2== Configure Bind DNS =2==
Goal: By the end of this, you should have two machines.  A broker machine, and one node machine.  You should be able to create applications, that will be put on the node machine.  You should be able to check the status of those applications.  You should be able to point your web browser to the URL of those applications.
yum -y install bind bind-utils


domain=example.com
Note: There is no web console in Fedora 19. That will be in Fedora 20.
keyfile=/var/named/${domain}.key
brokerip="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"


cd /var/named/
These instructions were created most from the following two places.
dnssec-keygen -a HMAC-MD5 -b 512 -n USER -r /dev/urandom ${domain}
*  https://www.openshift.com/wiki/build-your-own
KEY="$(grep Key: K${domain}*.private | cut -d ' ' -f 2)"
* https://www.openshift.com/forums/openshift/fedora-18-openshift-origin-setup-steps-and-testing
cd -
rndc-confgen -a -r /dev/urandom
echo $KEY


restorecon -v /etc/rndc.* /etc/named.*
= '''''Initial Setup of Broker and Node Machines''''' =
chown -v root:named /etc/rndc.key
chmod -v 640 /etc/rndc.key


echo "forwarders { 8.8.8.8; 8.8.4.4; } ;" >> /var/named/forwarders.conf
'''ON BOTH BROKER AND NODE'''
restorecon -v /var/named/forwarders.conf
chmod -v 755 /var/named/forwarders.conf


rm -rvf /var/named/dynamic
# Start with a Fedora 19 minimal install
mkdir -vp /var/named/dynamic
yum -y update
# avoid clock skew
yum -y install ntp
/bin/systemctl enable ntpd.service
/bin/systemctl start  ntpd.service


echo $domain
'''ON BROKER'''
export DOMAIN="example.com"
export BROKERIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"
export BROKERNAME="broker.example.com"
export NODEIP="--- IP Address from Node machine ---"
export NODENAME="node.example.com"
# Here is the IP Address from Broker machine
nm-tool | grep Address | grep -v HW | awk '{print $2}'


cat <<EOF > /var/named/dynamic/${domain}.db
'''ON NODE'''
\$ORIGIN .
export DOMAIN="example.com"
\$TTL 1 ; 1 seconds (for testing only)
export BROKERIP="--- IP Address from Broker machine ---"
${domain} IN SOA ns1.${domain}. hostmaster.${domain}. (
export BROKERNAME="broker.example.com"
                        2011112904 ; serial
export NODEIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"
                        60        ; refresh (1 minute)
export NODENAME="node.example.com"
                        15        ; retry (15 seconds)
# Here is the IP Address from Node machine
                        1800      ; expire (30 minutes)
nm-tool | grep Address | grep -v HW | awk '{print $2}'
                        10        ; minimum (10 seconds)
                          )
                    NS ns1.${domain}.
                    MX 10 mail.${domain}.
\$ORIGIN ${domain}.
ns1               A        127.0.0.1


EOF
= '''''Setup and Configure Broker''''' =


cat <<EOF > /var/named/${domain}.key
== '''Broker: Bind DNS''' ==
key ${domain} {
  algorithm HMAC-MD5;
  secret "${KEY}";
};
EOF


cat /var/named/dynamic/${domain}.db
yum -y install bind bind-utils
cat /var/named/${domain}.key
KEYFILE=/var/named/${DOMAIN}.key


chown -Rv named:named /var/named
setup DNSSEC key pair
restorecon -rv /var/named
cd /var/named/
dnssec-keygen -a HMAC-MD5 -b 512 -n USER -r /dev/urandom ${DOMAIN}
KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)"
cd -
rndc-confgen -a -r /dev/urandom
echo $KEY


mv /etc/named.conf /etc/named.conf.openshift
setup permissions for the DNSSEC key pair
cat <<EOF > /etc/named.conf
restorecon -v /etc/rndc.* /etc/named.*
// named.conf
chown -v root:named /etc/rndc.key
//
chmod -v 640 /etc/rndc.key
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//


options {
setup forwarders
    listen-on port 53 { any; };
echo "forwarders { 8.8.8.8; 8.8.4.4; } ;" >> /var/named/forwarders.conf
    directory "/var/named";
restorecon -v /var/named/forwarders.conf
    dump-file "/var/named/data/cache_dump.db";
chmod -v 755 /var/named/forwarders.conf
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    allow-query { any; };
    recursion yes;


    /* Path to ISC DLV key */
setup initial DNS database
    bindkeys-file "/etc/named.iscdlv.key";
rm -rvf /var/named/dynamic
 
mkdir -vp /var/named/dynamic
    // set forwarding to the next nearest server (from DHCP response
    forward only;
cat <<EOF > /var/named/dynamic/${DOMAIN}.db
    include "forwarders.conf";
\$ORIGIN .
};
\$TTL 1 ; 1 seconds (for testing only)
 
${DOMAIN} IN SOA ns1.${DOMAIN}. hostmaster.${DOMAIN}. (
logging {
                          2011112904 ; serial
    channel default_debug {
                          60         ; refresh (1 minute)
         file "data/named.run";
                          15         ; retry (15 seconds)
         severity dynamic;
                          1800      ; expire (30 minutes)
    };
                          10        ; minimum (10 seconds)
};
                          )
 
                      NS ns1.${DOMAIN}.
// use the default rndc key
                      MX 10 mail.${DOMAIN}.
include "/etc/rndc.key";
\$ORIGIN ${DOMAIN}.
ns1               A        127.0.0.1
   
   
controls {
EOF
    inet 127.0.0.1 port 953
    allow { 127.0.0.1; } keys { "rndc-key"; };
};


include "/etc/named.rfc1912.zones";
Install the DNSSEC key
cat <<EOF > ${KEYFILE}
key ${DOMAIN} {
  algorithm HMAC-MD5;
  secret "${KEY}";
};
EOF


include "${domain}.key";
Check the key and database
cat /var/named/dynamic/${DOMAIN}.db
cat /var/named/${DOMAIN}.key


zone "${domain}" IN {
Set permissions for key and database
    type master;
chown -Rv named:named /var/named
    file "dynamic/${domain}.db";
restorecon -rv /var/named
    allow-update { key ${domain} ; } ;
};
EOF


cat /etc/named.conf
Create the named configuration file
chown -v root:named /etc/named.conf
mv /etc/named.conf /etc/named.conf.openshift
restorecon /etc/named.conf
cat <<EOF > /etc/named.conf
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
options {
    listen-on port 53 { any; };
    directory "/var/named";
    dump-file "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    allow-query { any; };
    recursion yes;
    /* Path to ISC DLV key */
    bindkeys-file "/etc/named.iscdlv.key";
    // set forwarding to the next nearest server (from DHCP response)
    forward only;
    include "forwarders.conf";
};
logging {
    channel default_debug {
        file "data/named.run";
        severity dynamic;
    };
};
// use the default rndc key
include "/etc/rndc.key";
 
controls {
    inet 127.0.0.1 port 953
    allow { 127.0.0.1; } keys { "rndc-key"; };
};
include "/etc/named.rfc1912.zones";
include "${DOMAIN}.key";
zone "${DOMAIN}" IN {
    type master;
    file "dynamic/${DOMAIN}.db";
    allow-update { key ${DOMAIN} ; } ;
};
EOF


vi /etc/resolv.conf
Check the named config file
# Add the following as the first nameserver in the file
cat /etc/named.conf
# nameserver **your broker ip address**


firewall-cmd --add-service=dns
setup permissions of named config file
firewall-cmd --permanent --add-service=dns
chown -v root:named /etc/named.conf
firewall-cmd --list-all
restorecon /etc/named.conf
/bin/systemctl enable named.service
/bin/systemctl start named.service


nsupdate -k ${keyfile}
Setup firewall
> server 127.0.0.1
firewall-cmd --add-service=dns
> update delete broker.example.com A
firewall-cmd --permanent --add-service=dns
> update add broker.example.com 180 A **your broker ip address**
firewall-cmd --list-all
> send
> quit


ping broker.example.com
Setup and start service
dig @127.0.0.1 broker.example.com
/bin/systemctl enable named.service
/bin/systemctl start named.service


add entries using nsupdate
nsupdate -k ${KEYFILE}
> server 127.0.0.1
> update delete broker.example.com A
> update add **your broker full name ** 180 A **your broker ip address**
(example: update add broker.example.com 180 A 192.168.122.220 )
> send
> quit


=3== Configure the BROKER DHCP client and hostname =3==
Test DNS server
This is best done before hostname has been set.
ping broker.example.com
dig @127.0.0.1 broker.example.com


echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
== '''Broker: DHCP client and hostname''' ==
echo "supersede host-name \"broker\";" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf


echo "broker.example.com" > /etc/hostname
Setup dhcp client
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede host-name \"broker\";" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf


=4== Installing and configuring MongoDB =4==
Setup hostname
echo "broker.example.com" > /etc/hostname


yum -y install mongodb-server
== '''Broker: MongoDB''' ==
Install Software
yum -y install mongodb-server


vi /etc/mongodb.conf
Tweak config file
# Uncomment auth = true
vi /etc/mongodb.conf
# Add smallfiles = true
# Uncomment auth = true
# Add smallfiles = true


/usr/bin/systemctl enable mongod.service
Setup and start service
/usr/bin/systemctl status mongod.service
/usr/bin/systemctl enable mongod.service
/usr/bin/systemctl start mongod.service
/usr/bin/systemctl status mongod.service
/usr/bin/systemctl status mongod.service
/usr/bin/systemctl start mongod.service
/usr/bin/systemctl status mongod.service


# Testing
Testing
mongo
mongo
> show dbs
> show dbs
> exit
> exit


=5== Installing and configuring QPID =5==
== '''Broker: Messaging (using QPID)''' ==
# Activemq on F19 isn't ready for production.  When it is, we'll use that
Activemq on F19 isn't ready for OpenShift production.  When it is, we'll use that
# For now let's use QPID with mcollective.
For now we'll use QPID with mcollective.


yum install mcollective-qpid-plugin qpid-cpp-server
Install Software
firewall-cmd --add-port=5672/tcp
yum install mcollective-qpid-plugin qpid-cpp-server
firewall-cmd --permanent --add-port=5672/tcp
firewall-cmd --list-all


/usr/bin/systemctl enable qpidd.service
Setup Firewall
/usr/bin/systemctl start qpidd.service
firewall-cmd --add-port=5672/tcp
/usr/bin/systemctl status qpidd.service
firewall-cmd --permanent --add-port=5672/tcp
firewall-cmd --list-all


=6== Installing and configuring MCollective client (QPID) =6==
Setup and start service
yum -y install mcollective-client
/usr/bin/systemctl enable qpidd.service
mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig
/usr/bin/systemctl start qpidd.service
/usr/bin/systemctl status qpidd.service


cat <<EOF > /etc/mcollective/client.cfg
== '''Broker: MCollective client ( using QPID)''' ==
topicprefix = /topic/
Install Software
main_collective = mcollective
yum -y install mcollective-client
collectives = mcollective
libdir = /usr/libexec/mcollective
loglevel = debug
logfile = /var/log/mcollective-client.log


# Plugins
Move original config file out of the way
securityprovider = psk
mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig
plugin.psk = unset
connector = qpid
plugin.qpid.host=broker.example.com
plugin.qpid.secure=false
plugin.qpid.timeout=5


# Facts
Create new client config file.  This config file is for using QPID as a messaging platform.
factsource = yaml
cat <<EOF > /etc/mcollective/client.cfg
plugin.yaml = /etc/mcollective/facts.yaml
topicprefix = /topic/
EOF
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
loglevel = debug
logfile = /var/log/mcollective-client.log
# Plugins
securityprovider = psk
plugin.psk = unset
connector = qpid
plugin.qpid.host=${BROKERNAME}
plugin.qpid.secure=false
plugin.qpid.timeout=5
# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
EOF


=7== Installing and configuring the broker application =7==
== '''Broker: broker application''' ==
# When mcollective was updated to 2.2.3 it created a conflict with one of our components.
Install software
# We are working on fixing the conflict, but until then, do the following.
yum -y install openshift-origin-broker openshift-origin-broker-util rubygem-openshift-origin-auth-remote-user rubygem-openshift-origin-msg-broker-mcollective rubygem-openshift-origin-dns-bind
yumdownloader openshift-origin-msg-common
rpm -Uvh openshift-origin-msg-common-1.4.1-1.fc19.noarch.rpm --nodeps --force


yum -y install openshift-origin-broker openshift-origin-broker-util rubygem-openshift-origin-auth-remote-user rubygem-openshift-origin-msg-broker-mcollective rubygem-openshift-origin-dns-bind
Modify the broker proxy server name
sed -i -e "s/ServerName .*$/ServerName broker.example.com/" /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf
cat /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf


sed -i -e "s/ServerName .*$/ServerName broker.example.com/" /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf
Setup and start service
cat /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf
/usr/bin/systemctl enable httpd.service
/usr/bin/systemctl enable ntpd.service
/usr/bin/systemctl enable sshd.service


/usr/bin/systemctl enable httpd.service
Setup Firewall
/usr/bin/systemctl enable ntpd.service
firewall-cmd --add-service=ssh
/usr/bin/systemctl enable sshd.service
firewall-cmd --add-service=http
firewall-cmd --add-service=https
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all


firewall-cmd --add-service=ssh
Generate access key
firewall-cmd --add-service=http
openssl genrsa -out /etc/openshift/server_priv.pem 2048
firewall-cmd --add-service=https
openssl rsa -in /etc/openshift/server_priv.pem -pubout > /etc/openshift/server_pub.pem
firewall-cmd --permanent --add-service=ssh
ssh-keygen -t rsa -b 2048 -f ~/.ssh/rsync_id_rsa
firewall-cmd --permanent --add-service=http
cp -v ~/.ssh/rsync_id_rsa* /etc/openshift/
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all


openssl genrsa -out /etc/openshift/server_priv.pem 2048
Setup selinux boolean variables and set file contexts
openssl rsa -in /etc/openshift/server_priv.pem -pubout > /etc/openshift/server_pub.pem
setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_run_stickshift=on named_write_master_zones=on
ssh-keygen -t rsa -b 2048 -f ~/.ssh/rsync_id_rsa
fixfiles -R rubygem-passenger restore
cp -v ~/.ssh/rsync_id_rsa* /etc/openshift/
fixfiles -R mod_passenger restore
restorecon -rv /var/run
restorecon -rv /usr/share/gems/gems/passenger-*


setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_run_stickshift=on named_write_master_zones=on
Tweak broker config, if needed
fixfiles -R rubygem-passenger restore
vi /etc/openshift/broker.conf
fixfiles -R mod_passenger restore
# Might not have to do anything but make sure you have the following lines
restorecon -rv /var/run
CLOUD_DOMAIN="example.com"
restorecon -rv /usr/share/gems/gems/passenger-*
VALID_GEAR_SIZES="small,medium"


vi /etc/openshift/broker.conf
== '''Broker: broker plugins and MongoDB user accounts''' ==
# Might not have to do anything
Create config files from examples
CLOUD_DOMAIN="example.com"
cp /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf.example /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf
VALID_GEAR_SIZES="small,medium"


=8== Configuring the broker plugins and MongoDB user accounts =8==
Config the DNS plugin
cd /var/named/
KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)"
cat $KEYFILE
echo $KEY
cat <<EOF > /etc/openshift/plugins.d/openshift-origin-dns-bind.conf
BIND_SERVER="127.0.0.1"
BIND_PORT=53
BIND_KEYNAME="${DOMAIN}"
BIND_KEYVALUE="${KEY}"
BIND_ZONE="${DOMAIN}"
EOF


cp /usr/share/gems/gems/openshift-origin-auth-remote-user-*/conf/openshift-origin-auth-remote-user.conf.example /etc/openshift/plugins.d/openshift-origin-auth-remote-user.conf
Configure authentication plugin and add a user
cp /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf.example /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf
cp -v /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user-basic.conf.sample /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user.conf
htpasswd -c -b -s /etc/openshift/htpasswd demo demo
# Don't forget your password. <demo password>
cat /etc/openshift/htpasswd


domain=example.com
Add MongoDB account
keyfile=/var/named/${domain}.key
grep MONGO /etc/openshift/broker.conf
cd /var/named/
mongo openshift_broker_dev --eval 'db.addUser("openshift", "mooo")'
KEY="$(grep Key: K${domain}*.private | cut -d ' ' -f 2)"
# If you are going to change the username and/or password, change broker.conf
cat $keyfile
echo $KEY


cd /etc/openshift/plugins.d/
Bundle broker gems
cat <<EOF > openshift-origin-dns-bind.conf
yum -y install rubygem-psych rubygem-mocha
BIND_SERVER="127.0.0.1"
cd /var/www/openshift/broker
BIND_PORT=53
gem install mongoid
BIND_KEYNAME="${domain}"
bundle --local
BIND_KEYVALUE="${KEY}"
BIND_ZONE="${domain}"
EOF


#pushd /usr/share/selinux/packages/rubygem-openshift-origin-dns-bind/ && make -f /usr/share/selinux/devel/Makefile ; popd
Setup and start services
#semodule -i /usr/share/selinux/packages/rubygem-openshift-origin-dns-bind/dhcpnamedforward.pp
/usr/bin/systemctl enable openshift-broker.service
/usr/bin/systemctl start httpd.service
/usr/bin/systemctl start openshift-broker.service
/usr/bin/systemctl status openshift-broker.service


cp -v /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user-basic.conf.sample /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user.conf
Test basic broker service
htpasswd -c -b -s /etc/openshift/htpasswd demo demopassword
curl -k -u demo:demo https://localhost/broker/rest/api
# Don't forget your password. <demo password>
cat /etc/openshift/htpasswd


grep MONGO /etc/openshift/broker.conf
= '''''Setup and Configure Node''''' =
mongo openshift_broker_dev --eval 'db.addUser("openshift", "mooo")'
# If you are going to change the username and/or password, change broker.conf


yum -y install rubygem-psych
== '''Node: Initial setup/configure''' ==
cd /var/www/openshift/broker
'''ON BROKER'''
# This is being fixed, but for now do the following
KEYFILE=/var/named/${DOMAIN}.key
vi Gemfile
# remove minitest version
# add gem 'psych'
gem install mongoid
bundle --local


/usr/bin/systemctl enable openshift-broker.service
Register the node in DNS
/usr/bin/systemctl start httpd.service
oo-register-dns -h ${NODENAME} -d ${DOMAIN} -n ${NODEIP} -k ${KEYFILE}
/usr/bin/systemctl start openshift-broker.service
/usr/bin/systemctl status openshift-broker.service


curl -k -u demo:demopassword https://localhost/broker/rest/api
Copy the broker public key to node
scp /etc/openshift/rsync_id_rsa.pub root@${NODENAME}:/root/.ssh/


'''ON NODE'''
Put the brokers public key in root authorized keys
cat /root/.ssh/rsync_id_rsa.pub >> /root/.ssh/authorized_keys
rm -f /root/.ssh/rsync_id_rsa.pub


=9== Install and configure the web console =9==
'''ON BROKER'''
The web console is not in Fedora 19, skipping this step
Test to make sure we can login using our key
ssh -i /root/.ssh/rsync_id_rsa root@${NODENAME}
exit


=10== Initial setup/configure of the node host =10==
== '''Node: DHCP client and hostname''' ==
# ON NODE
Configure the dhcp settings
yum update
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
yum -y install ntp
echo "supersede host-name \"node\";" >> /etc/dhcp/dhclient-eth0.conf
/bin/systemctl enable ntpd.service
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf
/bin/systemctl start  ntpd.service


# Find out the node ip address
Set the hostname
nm-tool
echo "node.example.com" > /etc/hostname


# ON BROKER
== '''Node: MCollective'''  ==
domain=example.com
'''ON NODE'''
keyfile=/var/named/${domain}.key
Install Software
# Use the IP address from the node, found above
yum -y install openshift-origin-msg-node-mcollective mcollective-qpid-plugin
oo-register-dns -h node -d ${domain} -n 192.168.122.161 -k ${keyfile}


scp /etc/openshift/rsync_id_rsa.pub root@node.example.com:/root/.ssh/
Move original configuration out of the way
mv /etc/mcollective/server.cfg /etc/mcollective/server.cfg.orig


# ON NODE
Create new configuration
cat /root/.ssh/rsync_id_rsa.pub >> /root/.ssh/authorized_keys
cat <<EOF > /etc/mcollective/server.cfg
rm -f /root/.ssh/rsync_id_rsa.pub
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logfile = /var/log/mcollective.log
loglevel = debug
daemonize = 1
direct_addressing = n
# Plugins
securityprovider = psk
plugin.psk = unset
connector = qpid
plugin.qpid.host=${BROKERNAME}
plugin.qpid.secure=false
plugin.qpid.timeout=5
# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
EOF


# ON BROKER
Setup and start services
ssh -i /root/.ssh/rsync_id_rsa root@node.example.com
/bin/systemctl enable mcollective.service
exit
/bin/systemctl start  mcollective.service


#Find out the broker ip address
'''ON BROKER'''
nm-tool
mco ping
# node should show up on mco ping


=10a== Configure the NODE DHCP client and hostname =10a==
== '''Node: node application''' ==
# ON NODE
Install software
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util
echo "supersede host-name \"node\";" >> /etc/dhcp/dhclient-eth0.conf
yum -y install openshift-origin-cartridge-cron-1.4 openshift-origin-cartridge-diy-0.1
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf
echo "node.example.com" > /etc/hostname


reboot
Setup firewall
firewall-cmd --add-service=ssh
firewall-cmd --add-service=http
firewall-cmd --add-service=https
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all


=11== Setting up MCollective on the node host =11==
== '''Node: PAM namespace module, cgroups, and user quotas''' ==
# ON NODE
PAM
yum -y install openshift-origin-msg-node-mcollective
sed -i -e 's|pam_selinux|pam_openshift|g' /etc/pam.d/sshd
mv /etc/mcollective/server.cfg /etc/mcollective/server.cfg.orig
for f in "runuser" "runuser-l" "sshd" "su" "system-auth-ac"
do
  t="/etc/pam.d/$f"
  if ! grep -q "pam_namespace.so" "$t"
  then
    echo -e "session\t\trequired\tpam_namespace.so no_unmount_on_close" >> "$t"
  fi
done


cat <<EOF > /etc/mcollective/server.cfg
CGROUPS
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logfile = /var/log/mcollective.log
loglevel = debug
daemonize = 1
direct_addressing = n


# Plugins
Cgroups Config - Need to still fixup the cgroup configurations
securityprovider = psk
echo "mount {" >> /etc/cgconfig.conf
plugin.psk = unset
echo "        cpu    = /cgroup/all;" >> /etc/cgconfig.conf
connector = qpid
echo "        cpuacct = /cgroup/all;" >> /etc/cgconfig.conf
plugin.qpid.host=broker.example.com
echo "        memory  = /cgroup/all;" >> /etc/cgconfig.conf
plugin.qpid.secure=false
echo "        freezer = /cgroup/all;" >> /etc/cgconfig.conf
plugin.qpid.timeout=5
echo "        net_cls = /cgroup/all;" >> /etc/cgconfig.conf
echo "}" >> /etc/cgconfig.conf
restorecon -v /etc/cgconfig.conf
mkdir /cgroup
restorecon -RFvv /cgroup


# Facts
Cgroups enable and startup services
factsource = yaml
/bin/systemctl enable cgconfig.service
plugin.yaml = /etc/mcollective/facts.yaml
/bin/systemctl enable cgred.service
EOF
/usr/sbin/chkconfig openshift-cgroups on
/bin/systemctl restart  cgconfig.service
/bin/systemctl restart  cgred.service
/usr/sbin/service openshift-cgroups restart


/bin/systemctl enable mcollective.service
DISK QUOTA
/bin/systemctl start mcollective.service
# Edit fstab and add usrquota to whichever filesystem
#  has /var/lib/openshift on it
UUID=b9e21eae-4b8c-4936-9f5d-d10631ff535e / ext4    defaults,usrquota 1 1
# reboot or remount
mount -o remount /
  quotacheck -cmug /


# ON BROKER
== '''Node: SELinux and System Control''' ==
mco ping
Setup SELINUX Booleans
setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_read_user_content=on httpd_enable_homedirs=on httpd_run_stickshift=on allow_polyinstantiation=on


=12== Setting up node packages on the node host =12==
Update selinux file setting
# ON NODE
restorecon -rv /var/run
yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util
restorecon -rv /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid
yum -y install openshift-origin-cartridge-cron-1.4 openshift-origin-cartridge-diy-0.1
restorecon -rv /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift


firewall-cmd --add-service=ssh
SYSTEM CONTROL SETTINGS
firewall-cmd --add-service=http
echo "# Added for OpenShift" >> /etc/sysctl.d/openshift.conf
firewall-cmd --add-service=https
echo "kernel.sem = 250  32000 32  4096" >> /etc/sysctl.d/openshift.conf
firewall-cmd --permanent --add-service=ssh
echo "net.ipv4.ip_local_port_range = 15000 35530" >> /etc/sysctl.d/openshift.conf
firewall-cmd --permanent --add-service=http
echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/openshift.conf
firewall-cmd --permanent --add-service=https
sysctl -p /etc/sysctl.d/openshift.conf
firewall-cmd --list-all


=13== Configuring PAM namespace module, cgropus, and user quotas on the node host =13==
== '''Node: SSH, Port Proxy, and Node application''' ==
# ON NODE
SSH
# PAM
vi /etc/ssh/sshd_config
sed -i -e 's|pam_selinux|pam_openshift|g' /etc/pam.d/sshd
> AcceptEnv GIT_SSH
perl -p -i -e "s/^#MaxSessions .*$/MaxSessions 40/" /etc/ssh/sshd_config
perl -p -i -e "s/^#MaxStartups .*$/MaxStartups 40/" /etc/ssh/sshd_config
/bin/systemctl restart  sshd.service


for f in "runuser" "runuser-l" "sshd" "su" "system-auth-ac"
PORT PROXY
do
firewall-cmd --add-port=35531-65535/tcp
  t="/etc/pam.d/$f"
firewall-cmd --permanent --add-port=35531-65535/tcp
  if ! grep -q "pam_namespace.so" "$t"
firewall-cmd --list-all
  then
    echo -e "session\t\trequired\tpam_namespace.so no_unmount_on_close" >> "$t"
/bin/systemctl enable openshift-port-proxy.service
  fi
/bin/systemctl restart  openshift-port-proxy.service
done


#CGROUPS
NODE SETUP
#echo "mount {" >> /etc/cgconfig.conf
/bin/systemctl enable httpd.service
#echo "        cpu    = /cgroup/all;" >> /etc/cgconfig.conf
/bin/systemctl enable openshift-gears.service
#echo "        cpuacct = /cgroup/all;" >> /etc/cgconfig.conf
#echo "        memory = /cgroup/all;" >> /etc/cgconfig.conf
vi /etc/openshift/node.conf
#echo "        freezer = /cgroup/all;" >> /etc/cgconfig.conf
> PUBLIC_HOSTNAME="node.example.com"
#echo "        net_cls = /cgroup/all;" >> /etc/cgconfig.conf
  > PUBLIC_IP="192.168.122.161" (Node IP Address)
#echo "}" >> /etc/cgconfig.conf
> BROKER_HOST="192.168.122.220" (Broker IP Address)
#restorecon -v /etc/cgconfig.conf
> CLOUD_DOMAIN="example.com"
#mkdir /cgroup
#restorecon -RFvv /cgroup
/etc/cron.minutely/openshift-facts


/bin/systemctl enable cgconfig.service
== '''Node: Reboot''' ==
/bin/systemctl enable cgred.service
We need to reboot to load all the node stuff correctly
/usr/sbin/chkconfig openshift-cgroups on
  reboot
/bin/systemctl restart  cgconfig.service
/bin/systemctl restart cgred.service
/usr/sbin/service openshift-cgroups restart


#DISK QUOTA
= '''''Testing''''' =
# Edit fstab and add usrquota to whichever filesystem
==Test on Broker (after node is back up)==
#  has /var/lib/openshift on it
UUID=b9e21eae-4b8c-4936-9f5d-d10631ff535e / ext4    defaults,usrquota 1 1
# reboot or remount
mount -o remount /
quotacheck -cmug /


=14== Configuring SELinux and System Control on the node host =14==
'''Check Messaging'''
# ON NODE
mco ping
# SELINUX
Should look like
setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_read_user_content=on httpd_enable_homedirs=on httpd_run_stickshift=on allow_polyinstantiation=on
node.example.com                        time=239.51 ms
---- ping statistics ----
1 replies max: 239.51 min: 239.51 avg: 239.51


restorecon -rv /var/run
'''Check Broker'''
restorecon -rv /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid
curl -k -u demo:demo https://localhost/broker/rest/api
restorecon -rv /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift
Should look like
{"data":{"API":{"href":"https://localhost/broker/rest/api","method":"GET","optional_params":[],"rel":"API entry point","required_params":[]},"GET_ENVIRONMENT":{"href":"https://localhost/broker/rest/environment","method":"GET","optional_params":[],"rel":"Get environment information","required_params":[]},"GET_USER"
...
:id","type":"string","valid_options":[]}]}},"messages":[],"status":"ok","supported_api_versions":[1.0,1.1,1.2,1.3],"type":"links","version":"1.3"}


# SYSTEM CONTROL SETTINGS
'''Check and Setup User'''
echo "# Added for OpenShift" >> /etc/sysctl.d/openshift.conf
yum -y install rubygem-rhc
echo "kernel.sem = 250 32000 32 4096" >> /etc/sysctl.d/openshift.conf
LIBRA_SERVER=broker.example.com rhc setup
echo "net.ipv4.ip_local_port_range = 15000 35530" >> /etc/sysctl.d/openshift.conf
Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot)
echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/openshift.conf
OpenShift Client Tools (RHC) Setup Wizard
sysctl -p /etc/sysctl.d/openshift.conf
This wizard will help you upload your SSH keys, set your application namespace, and
check that other programs like Git are properly installed.
The server's certificate is self-signed, which means that a secure connection can't be
established to 'broker.example.com'.
You may bypass this check, but any data you send to the server could be intercepted by
others.
Connect without checking the certificate? (yes|no): yes
Login to broker.example.com: demo
Password: ****
 
OpenShift can create and store a token on disk which allows to you to access the
server without using your password. The key is stored in your home directory and
should be kept secret.  You can delete the key at any time by running 'rhc logout'.
Generate a token now? (yes|no) no
   
  Saving configuration to /root/.openshift/express.conf ... done
No SSH keys were found. We will generate a pair of keys for you.
    Created: /root/.ssh/id_rsa.pub
Your public SSH key must be uploaded to the OpenShift server to access code.  Upload
now? (yes|no)
yes
Since you do not have any keys associated with your OpenShift account, your new key
will be uploaded as the 'default' key.
Uploading key 'default' ... done
Checking for git ... found git version 1.8.2.1
Checking common problems .. done
Checking your namespace ... none
Your namespace is unique to your account and is the suffix of the public URLs we
assign to your applications. You may configure your namespace here or leave it blank
and use 'rhc create-domain' to create a namespace later. You will not be able to
create applications without first creating a namespace.
Please enter a namespace (letters and numbers only) |<none>|: demoland


=14== Configuring SSH, Port Proxy, and Node on the node host =14==
Create an app
# ON NODE
rhc domain show -p demo
# SSH
rhc app create test1 diy-0.1 -p demo
vi /etc/ssh/sshd_config
> AcceptEnv GIT_SSH


perl -p -i -e "s/^#MaxSessions .*$/MaxSessions 40/" /etc/ssh/sshd_config
==Test on Local Machine (after node is back up)==
perl -p -i -e "s/^#MaxStartups .*$/MaxStartups 40/" /etc/ssh/sshd_config
Setup your machine to use broker as a name server (Note: This might mess up normal network operations.)
vi /etc/resolve.conf
# At the first line put "nameserver *broker ip address*"
nameserver 192.168.122.220


/bin/systemctl restart sshd.service
'''Check and Setup User'''
 
yum -y install rubygem-rhc
# PORT PROXY
  LIBRA_SERVER=broker.example.com rhc setup
 
Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot)
firewall-cmd --add-port=35531-65535/tcp
OpenShift Client Tools (RHC) Setup Wizard
firewall-cmd --permanent --add-port=35531-65535/tcp
firewall-cmd --list-all
This wizard will help you upload your SSH keys, set your application namespace, and
 
check that other programs like Git are properly installed.
/bin/systemctl enable openshift-port-proxy.service
/bin/systemctl restart openshift-port-proxy.service
The server's certificate is self-signed, which means that a secure connection can't be
 
established to 'broker.example.com'.
# NODE SETUP
/bin/systemctl enable openshift-gears.service
You may bypass this check, but any data you send to the server could be intercepted by
 
  others.
# Find node and broker IP address
nm-tool
Connect without checking the certificate? (yes|no): yes
 
Login to broker.example.com: demo
vi /etc/openshift/node.conf
Password: ****
> PUBLIC_HOSTNAME="node.example.com"
 
> PUBLIC_IP="192.168.122.161" (Node IP Address)
OpenShift can create and store a token on disk which allows to you to access the
> BROKER_HOST="192.168.122.220" (Broker IP Address)
server without using your password. The key is stored in your home directory and
> CLOUD_DOMAIN="example.com"
should be kept secret.  You can delete the key at any time by running 'rhc logout'.
Generate a token now? (yes|no) no
Saving configuration to /root/.openshift/express.conf ... done
No SSH keys were found. We will generate a pair of keys for you.
    Created: /root/.ssh/id_rsa.pub
Your public SSH key must be uploaded to the OpenShift server to access code. Upload
now? (yes|no)
yes
Since you do not have any keys associated with your OpenShift account, your new key
will be uploaded as the 'default' key.
Uploading key 'default' ... done
Checking for git ... found git version 1.8.2.1
Checking common problems .. done
Checking your namespace ... none
Your namespace is unique to your account and is the suffix of the public URLs we
assign to your applications. You may configure your namespace here or leave it blank
and use 'rhc create-domain' to create a namespace later. You will not be able to
create applications without first creating a namespace.
Please enter a namespace (letters and numbers only) |<none>|: demoland


/etc/cron.minutely/openshift-facts
Create an app
rhc domain show -p demo
rhc app create test2 diy-0.1 -p demo


=14== Reboot Node and test =14==
# ON NODE
reboot


# ON BROKER (after node is back up)
'''Check App'''
mco ping
You should be able to go to the following URL in your web browser.
curl -k -u demo:demo https://localhost/broker/rest/api


yum -y install rubygem-rhc
http://test2-demoland.example.com/
LIBRA_SERVER=broker.example.com rhc setup

Latest revision as of 13:39, 8 August 2013

Fedora 19 is when OpenShift Origin first became a feature.

NOTE: (August 8, 2013) This page is getting an update. It will accommodate F19 cloud images (not just minimal install). It is also updated with the OpenShift Origin Version 2 documentation.

This page is here to show how to setup OpenShift Origin on Fedora 19 using the packages in Fedora, as opposed to the packages published from upstream. These steps are written out to be done by hand. Yes, people can script and/or puppetize these steps. But these are written out so that people can see, and fine tune them.

Note: And now they have been written into scripts. https://github.com/tdawson/oo-install-scripts

Goal: By the end of this, you should have two machines. A broker machine, and one node machine. You should be able to create applications, that will be put on the node machine. You should be able to check the status of those applications. You should be able to point your web browser to the URL of those applications.

Note: There is no web console in Fedora 19. That will be in Fedora 20.

These instructions were created most from the following two places.

Initial Setup of Broker and Node Machines

ON BOTH BROKER AND NODE

# Start with a Fedora 19 minimal install
yum -y update
# avoid clock skew
yum -y install ntp
/bin/systemctl enable ntpd.service
/bin/systemctl start  ntpd.service

ON BROKER

export DOMAIN="example.com"
export BROKERIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"
export BROKERNAME="broker.example.com"
export NODEIP="--- IP Address from Node machine ---"
export NODENAME="node.example.com"
# Here is the IP Address from Broker machine
nm-tool | grep Address | grep -v HW | awk '{print $2}'

ON NODE

export DOMAIN="example.com"
export BROKERIP="--- IP Address from Broker machine ---"
export BROKERNAME="broker.example.com"
export NODEIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"
export NODENAME="node.example.com"
# Here is the IP Address from Node machine
nm-tool | grep Address | grep -v HW | awk '{print $2}'

Setup and Configure Broker

Broker: Bind DNS

yum -y install bind bind-utils

KEYFILE=/var/named/${DOMAIN}.key

setup DNSSEC key pair

cd /var/named/
dnssec-keygen -a HMAC-MD5 -b 512 -n USER -r /dev/urandom ${DOMAIN}
KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)"
cd -
rndc-confgen -a -r /dev/urandom
echo $KEY

setup permissions for the DNSSEC key pair

restorecon -v /etc/rndc.* /etc/named.*
chown -v root:named /etc/rndc.key
chmod -v 640 /etc/rndc.key

setup forwarders

echo "forwarders { 8.8.8.8; 8.8.4.4; } ;" >> /var/named/forwarders.conf
restorecon -v /var/named/forwarders.conf
chmod -v 755 /var/named/forwarders.conf

setup initial DNS database

rm -rvf /var/named/dynamic
mkdir -vp /var/named/dynamic

cat <<EOF > /var/named/dynamic/${DOMAIN}.db
\$ORIGIN .
\$TTL 1	; 1 seconds (for testing only)
${DOMAIN} IN SOA ns1.${DOMAIN}. hostmaster.${DOMAIN}. (
                         2011112904 ; serial
                         60         ; refresh (1 minute)
                         15         ; retry (15 seconds)
                         1800       ; expire (30 minutes)
                         10         ; minimum (10 seconds)
                          )
                     NS ns1.${DOMAIN}.
                     MX 10 mail.${DOMAIN}.
\$ORIGIN ${DOMAIN}.
ns1	              A        127.0.0.1

EOF

Install the DNSSEC key

cat <<EOF > ${KEYFILE}
key ${DOMAIN} {
  algorithm HMAC-MD5;
  secret "${KEY}";
};
EOF

Check the key and database

cat /var/named/dynamic/${DOMAIN}.db
cat /var/named/${DOMAIN}.key

Set permissions for key and database

chown -Rv named:named /var/named
restorecon -rv /var/named

Create the named configuration file

mv /etc/named.conf /etc/named.conf.openshift
cat <<EOF > /etc/named.conf
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//

options {
    listen-on port 53 { any; };
    directory "/var/named";
    dump-file "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    allow-query { any; };
    recursion yes;

    /* Path to ISC DLV key */
    bindkeys-file "/etc/named.iscdlv.key";

    // set forwarding to the next nearest server (from DHCP response)
    forward only;
    include "forwarders.conf";
};

logging {
    channel default_debug {
        file "data/named.run";
        severity dynamic;
    };
};

// use the default rndc key
include "/etc/rndc.key";
 
controls {
    inet 127.0.0.1 port 953
    allow { 127.0.0.1; } keys { "rndc-key"; };
};

include "/etc/named.rfc1912.zones";

include "${DOMAIN}.key";

zone "${DOMAIN}" IN {
    type master;
    file "dynamic/${DOMAIN}.db";
    allow-update { key ${DOMAIN} ; } ;
};
EOF

Check the named config file

cat /etc/named.conf

setup permissions of named config file

chown -v root:named /etc/named.conf
restorecon /etc/named.conf

Setup firewall

firewall-cmd --add-service=dns
firewall-cmd --permanent --add-service=dns
firewall-cmd --list-all

Setup and start service

/bin/systemctl enable named.service
/bin/systemctl start named.service

add entries using nsupdate

nsupdate -k ${KEYFILE}
> server 127.0.0.1
> update delete broker.example.com A
> update add **your broker full name ** 180 A **your broker ip address**
(example: update add broker.example.com 180 A 192.168.122.220 )
> send
> quit

Test DNS server This is best done before hostname has been set.

ping broker.example.com
dig @127.0.0.1 broker.example.com

Broker: DHCP client and hostname

Setup dhcp client

echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede host-name \"broker\";" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf

Setup hostname

echo "broker.example.com" > /etc/hostname

Broker: MongoDB

Install Software

yum -y install mongodb-server

Tweak config file

vi /etc/mongodb.conf
# Uncomment auth = true
# Add smallfiles = true

Setup and start service

/usr/bin/systemctl enable mongod.service
/usr/bin/systemctl status mongod.service
/usr/bin/systemctl start mongod.service
/usr/bin/systemctl status mongod.service

Testing

mongo
> show dbs
> exit

Broker: Messaging (using QPID)

Activemq on F19 isn't ready for OpenShift production. When it is, we'll use that For now we'll use QPID with mcollective.

Install Software

yum install mcollective-qpid-plugin qpid-cpp-server

Setup Firewall

firewall-cmd --add-port=5672/tcp
firewall-cmd --permanent --add-port=5672/tcp
firewall-cmd --list-all

Setup and start service

/usr/bin/systemctl enable qpidd.service
/usr/bin/systemctl start qpidd.service
/usr/bin/systemctl status qpidd.service

Broker: MCollective client ( using QPID)

Install Software

yum -y install mcollective-client

Move original config file out of the way

mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig

Create new client config file. This config file is for using QPID as a messaging platform.

cat <<EOF > /etc/mcollective/client.cfg
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
loglevel = debug
logfile = /var/log/mcollective-client.log

# Plugins
securityprovider = psk
plugin.psk = unset
connector = qpid
plugin.qpid.host=${BROKERNAME}
plugin.qpid.secure=false
plugin.qpid.timeout=5

# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
EOF

Broker: broker application

Install software

yum -y install openshift-origin-broker openshift-origin-broker-util rubygem-openshift-origin-auth-remote-user rubygem-openshift-origin-msg-broker-mcollective rubygem-openshift-origin-dns-bind

Modify the broker proxy server name

sed -i -e "s/ServerName .*$/ServerName broker.example.com/" /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf 
cat /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf

Setup and start service

/usr/bin/systemctl enable httpd.service
/usr/bin/systemctl enable ntpd.service
/usr/bin/systemctl enable sshd.service

Setup Firewall

firewall-cmd --add-service=ssh
firewall-cmd --add-service=http
firewall-cmd --add-service=https
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all

Generate access key

openssl genrsa -out /etc/openshift/server_priv.pem 2048
openssl rsa -in /etc/openshift/server_priv.pem -pubout > /etc/openshift/server_pub.pem
ssh-keygen -t rsa -b 2048 -f ~/.ssh/rsync_id_rsa
cp -v ~/.ssh/rsync_id_rsa* /etc/openshift/

Setup selinux boolean variables and set file contexts

setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_run_stickshift=on named_write_master_zones=on
fixfiles -R rubygem-passenger restore
fixfiles -R mod_passenger restore
restorecon -rv /var/run
restorecon -rv /usr/share/gems/gems/passenger-*

Tweak broker config, if needed

vi /etc/openshift/broker.conf
# Might not have to do anything but make sure you have the following lines
CLOUD_DOMAIN="example.com"
VALID_GEAR_SIZES="small,medium"

Broker: broker plugins and MongoDB user accounts

Create config files from examples

cp /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf.example /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf

Config the DNS plugin

cd /var/named/
KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)"
cat $KEYFILE
echo $KEY

cat <<EOF > /etc/openshift/plugins.d/openshift-origin-dns-bind.conf
BIND_SERVER="127.0.0.1"
BIND_PORT=53
BIND_KEYNAME="${DOMAIN}"
BIND_KEYVALUE="${KEY}"
BIND_ZONE="${DOMAIN}"
EOF

Configure authentication plugin and add a user

cp -v /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user-basic.conf.sample /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user.conf
htpasswd -c -b -s /etc/openshift/htpasswd demo demo
# Don't forget your password. <demo password>
cat /etc/openshift/htpasswd

Add MongoDB account

grep MONGO /etc/openshift/broker.conf
mongo openshift_broker_dev --eval 'db.addUser("openshift", "mooo")'
# If you are going to change the username and/or password, change broker.conf

Bundle broker gems

yum -y install rubygem-psych rubygem-mocha
cd /var/www/openshift/broker
gem install mongoid
bundle --local

Setup and start services

/usr/bin/systemctl enable openshift-broker.service
/usr/bin/systemctl start httpd.service
/usr/bin/systemctl start openshift-broker.service
/usr/bin/systemctl status openshift-broker.service

Test basic broker service

curl -k -u demo:demo https://localhost/broker/rest/api

Setup and Configure Node

Node: Initial setup/configure

ON BROKER

KEYFILE=/var/named/${DOMAIN}.key

Register the node in DNS

oo-register-dns -h ${NODENAME} -d ${DOMAIN} -n ${NODEIP} -k ${KEYFILE}

Copy the broker public key to node

scp /etc/openshift/rsync_id_rsa.pub root@${NODENAME}:/root/.ssh/

ON NODE Put the brokers public key in root authorized keys

cat /root/.ssh/rsync_id_rsa.pub >> /root/.ssh/authorized_keys
rm -f /root/.ssh/rsync_id_rsa.pub

ON BROKER Test to make sure we can login using our key

ssh -i /root/.ssh/rsync_id_rsa root@${NODENAME}
exit

Node: DHCP client and hostname

Configure the dhcp settings

echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede host-name \"node\";" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf

Set the hostname

echo "node.example.com" > /etc/hostname

Node: MCollective

ON NODE Install Software

yum -y install openshift-origin-msg-node-mcollective mcollective-qpid-plugin

Move original configuration out of the way

mv /etc/mcollective/server.cfg /etc/mcollective/server.cfg.orig

Create new configuration

cat <<EOF > /etc/mcollective/server.cfg
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logfile = /var/log/mcollective.log
loglevel = debug
daemonize = 1
direct_addressing = n

# Plugins
securityprovider = psk
plugin.psk = unset
connector = qpid
plugin.qpid.host=${BROKERNAME}
plugin.qpid.secure=false
plugin.qpid.timeout=5

# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
EOF

Setup and start services

/bin/systemctl enable mcollective.service
/bin/systemctl start  mcollective.service

ON BROKER

mco ping
# node should show up on mco ping

Node: node application

Install software

yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util
yum -y install openshift-origin-cartridge-cron-1.4 openshift-origin-cartridge-diy-0.1

Setup firewall

firewall-cmd --add-service=ssh
firewall-cmd --add-service=http
firewall-cmd --add-service=https
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all

Node: PAM namespace module, cgroups, and user quotas

PAM

sed -i -e 's|pam_selinux|pam_openshift|g' /etc/pam.d/sshd

for f in "runuser" "runuser-l" "sshd" "su" "system-auth-ac"
do
  t="/etc/pam.d/$f"
  if ! grep -q "pam_namespace.so" "$t"
  then
    echo -e "session\t\trequired\tpam_namespace.so no_unmount_on_close" >> "$t"
  fi
done

CGROUPS

Cgroups Config - Need to still fixup the cgroup configurations

echo "mount {" >> /etc/cgconfig.conf
echo "        cpu     = /cgroup/all;" >> /etc/cgconfig.conf
echo "        cpuacct = /cgroup/all;" >> /etc/cgconfig.conf
echo "        memory  = /cgroup/all;" >> /etc/cgconfig.conf
echo "        freezer = /cgroup/all;" >> /etc/cgconfig.conf
echo "        net_cls = /cgroup/all;" >> /etc/cgconfig.conf
echo "}" >> /etc/cgconfig.conf
restorecon -v /etc/cgconfig.conf
mkdir /cgroup
restorecon -RFvv /cgroup

Cgroups enable and startup services

/bin/systemctl enable cgconfig.service
/bin/systemctl enable cgred.service
/usr/sbin/chkconfig openshift-cgroups on
/bin/systemctl restart  cgconfig.service
/bin/systemctl restart  cgred.service
/usr/sbin/service openshift-cgroups restart

DISK QUOTA

# Edit fstab and add usrquota to whichever filesystem 
#   has /var/lib/openshift on it
UUID=b9e21eae-4b8c-4936-9f5d-d10631ff535e / ext4    defaults,usrquota 1 1
# reboot or remount
mount -o remount /
quotacheck -cmug /

Node: SELinux and System Control

Setup SELINUX Booleans

setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_read_user_content=on httpd_enable_homedirs=on httpd_run_stickshift=on allow_polyinstantiation=on

Update selinux file setting

restorecon -rv /var/run
restorecon -rv /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid
restorecon -rv /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift

SYSTEM CONTROL SETTINGS

echo "# Added for OpenShift" >> /etc/sysctl.d/openshift.conf
echo "kernel.sem = 250  32000 32  4096" >> /etc/sysctl.d/openshift.conf
echo "net.ipv4.ip_local_port_range = 15000 35530" >> /etc/sysctl.d/openshift.conf
echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/openshift.conf
sysctl -p /etc/sysctl.d/openshift.conf

Node: SSH, Port Proxy, and Node application

SSH

vi /etc/ssh/sshd_config
> AcceptEnv GIT_SSH

perl -p -i -e "s/^#MaxSessions .*$/MaxSessions 40/" /etc/ssh/sshd_config
perl -p -i -e "s/^#MaxStartups .*$/MaxStartups 40/" /etc/ssh/sshd_config

/bin/systemctl restart  sshd.service

PORT PROXY

firewall-cmd --add-port=35531-65535/tcp
firewall-cmd --permanent --add-port=35531-65535/tcp
firewall-cmd --list-all

/bin/systemctl enable openshift-port-proxy.service
/bin/systemctl restart  openshift-port-proxy.service

NODE SETUP

/bin/systemctl enable httpd.service
/bin/systemctl enable openshift-gears.service

vi /etc/openshift/node.conf
> PUBLIC_HOSTNAME="node.example.com"
> PUBLIC_IP="192.168.122.161" (Node IP Address)
> BROKER_HOST="192.168.122.220" (Broker IP Address)
> CLOUD_DOMAIN="example.com"

/etc/cron.minutely/openshift-facts

Node: Reboot

We need to reboot to load all the node stuff correctly

reboot

Testing

Test on Broker (after node is back up)

Check Messaging

mco ping

Should look like

node.example.com                         time=239.51 ms

---- ping statistics ----
1 replies max: 239.51 min: 239.51 avg: 239.51 

Check Broker

curl -k -u demo:demo https://localhost/broker/rest/api

Should look like

{"data":{"API":{"href":"https://localhost/broker/rest/api","method":"GET","optional_params":[],"rel":"API entry point","required_params":[]},"GET_ENVIRONMENT":{"href":"https://localhost/broker/rest/environment","method":"GET","optional_params":[],"rel":"Get environment information","required_params":[]},"GET_USER"
...
:id","type":"string","valid_options":[]}]}},"messages":[],"status":"ok","supported_api_versions":[1.0,1.1,1.2,1.3],"type":"links","version":"1.3"}

Check and Setup User

yum -y install rubygem-rhc
LIBRA_SERVER=broker.example.com rhc setup

Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot)

OpenShift Client Tools (RHC) Setup Wizard

This wizard will help you upload your SSH keys, set your application namespace, and
check that other programs like Git are properly installed.

The server's certificate is self-signed, which means that a secure connection can't be
established to 'broker.example.com'.

You may bypass this check, but any data you send to the server could be intercepted by
others.

Connect without checking the certificate? (yes|no): yes
Login to broker.example.com: demo
Password: ****
 
OpenShift can create and store a token on disk which allows to you to access the
server without using your password. The key is stored in your home directory and
should be kept secret.  You can delete the key at any time by running 'rhc logout'.
Generate a token now? (yes|no) no

Saving configuration to /root/.openshift/express.conf ... done

No SSH keys were found. We will generate a pair of keys for you.

    Created: /root/.ssh/id_rsa.pub

Your public SSH key must be uploaded to the OpenShift server to access code.  Upload
now? (yes|no)
yes

Since you do not have any keys associated with your OpenShift account, your new key
will be uploaded as the 'default' key.

Uploading key 'default' ... done

Checking for git ... found git version 1.8.2.1

Checking common problems .. done

Checking your namespace ... none

Your namespace is unique to your account and is the suffix of the public URLs we
assign to your applications. You may configure your namespace here or leave it blank
and use 'rhc create-domain' to create a namespace later.  You will not be able to
create applications without first creating a namespace.

Please enter a namespace (letters and numbers only) |<none>|: demoland

Create an app

rhc domain show -p demo
rhc app create test1 diy-0.1 -p demo

Test on Local Machine (after node is back up)

Setup your machine to use broker as a name server (Note: This might mess up normal network operations.)

vi /etc/resolve.conf
# At the first line put "nameserver *broker ip address*"
nameserver 192.168.122.220

Check and Setup User

yum -y install rubygem-rhc
LIBRA_SERVER=broker.example.com rhc setup

Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot)

OpenShift Client Tools (RHC) Setup Wizard

This wizard will help you upload your SSH keys, set your application namespace, and
check that other programs like Git are properly installed.

The server's certificate is self-signed, which means that a secure connection can't be
established to 'broker.example.com'.

You may bypass this check, but any data you send to the server could be intercepted by
others.

Connect without checking the certificate? (yes|no): yes
Login to broker.example.com: demo
Password: ****
 
OpenShift can create and store a token on disk which allows to you to access the
server without using your password. The key is stored in your home directory and
should be kept secret.  You can delete the key at any time by running 'rhc logout'.
Generate a token now? (yes|no) no

Saving configuration to /root/.openshift/express.conf ... done

No SSH keys were found. We will generate a pair of keys for you.

    Created: /root/.ssh/id_rsa.pub

Your public SSH key must be uploaded to the OpenShift server to access code.  Upload
now? (yes|no)
yes

Since you do not have any keys associated with your OpenShift account, your new key
will be uploaded as the 'default' key.

Uploading key 'default' ... done

Checking for git ... found git version 1.8.2.1

Checking common problems .. done

Checking your namespace ... none

Your namespace is unique to your account and is the suffix of the public URLs we
assign to your applications. You may configure your namespace here or leave it blank
and use 'rhc create-domain' to create a namespace later.  You will not be able to
create applications without first creating a namespace.

Please enter a namespace (letters and numbers only) |<none>|: demoland

Create an app

rhc domain show -p demo
rhc app create test2 diy-0.1 -p demo


Check App You should be able to go to the following URL in your web browser.

http://test2-demoland.example.com/