From Fedora Project Wiki

No edit summary
No edit summary
 
(38 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Fedora 19 is when OpenShift Origin first became a feature.
Fedora 19 is when OpenShift Origin first became a feature.
NOTE: (August 8, 2013) This page is getting an update.  It will accommodate F19 cloud images (not just minimal install).  It is also updated with the OpenShift Origin Version 2 documentation.


This page is here to show how to setup OpenShift Origin on Fedora 19 using the packages in Fedora, as opposed to the packages published from upstream.  These steps are written out to be done by hand.  Yes, people can script and/or puppetize these steps.  But these are written out so that people can see, and fine tune them.
This page is here to show how to setup OpenShift Origin on Fedora 19 using the packages in Fedora, as opposed to the packages published from upstream.  These steps are written out to be done by hand.  Yes, people can script and/or puppetize these steps.  But these are written out so that people can see, and fine tune them.
Note: And now they have been written into scripts. https://github.com/tdawson/oo-install-scripts


Goal: By the end of this, you should have two machines.  A broker machine, and one node machine.  You should be able to create applications, that will be put on the node machine.  You should be able to check the status of those applications.  You should be able to point your web browser to the URL of those applications.
Goal: By the end of this, you should have two machines.  A broker machine, and one node machine.  You should be able to create applications, that will be put on the node machine.  You should be able to check the status of those applications.  You should be able to point your web browser to the URL of those applications.
Line 11: Line 15:
* https://www.openshift.com/forums/openshift/fedora-18-openshift-origin-setup-steps-and-testing
* https://www.openshift.com/forums/openshift/fedora-18-openshift-origin-setup-steps-and-testing


= Initial Setup of Broker and Node Machines =
= '''''Initial Setup of Broker and Node Machines''''' =


'''ON BOTH BROKER AND NODE'''
'''ON BOTH BROKER AND NODE'''
Line 22: Line 26:
  /bin/systemctl start  ntpd.service
  /bin/systemctl start  ntpd.service


= Setup and Configure Broker =
'''ON BROKER'''
== Configure Bind DNS ==
export DOMAIN="example.com"
yum -y install bind bind-utils
export BROKERIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"
export BROKERNAME="broker.example.com"
export NODEIP="--- IP Address from Node machine ---"
export NODENAME="node.example.com"
# Here is the IP Address from Broker machine
nm-tool | grep Address | grep -v HW | awk '{print $2}'


domain=example.com
'''ON NODE'''
keyfile=/var/named/${domain}.key
export DOMAIN="example.com"
brokerip="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"
export BROKERIP="--- IP Address from Broker machine ---"
export BROKERNAME="broker.example.com"
export NODEIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"
export NODENAME="node.example.com"
# Here is the IP Address from Node machine
nm-tool | grep Address | grep -v HW | awk '{print $2}'


cd /var/named/
= '''''Setup and Configure Broker''''' =
dnssec-keygen -a HMAC-MD5 -b 512 -n USER -r /dev/urandom ${domain}
KEY="$(grep Key: K${domain}*.private | cut -d ' ' -f 2)"
cd -
rndc-confgen -a -r /dev/urandom
echo $KEY


restorecon -v /etc/rndc.* /etc/named.*
== '''Broker: Bind DNS''' ==
chown -v root:named /etc/rndc.key
chmod -v 640 /etc/rndc.key


echo "forwarders { 8.8.8.8; 8.8.4.4; } ;" >> /var/named/forwarders.conf
yum -y install bind bind-utils
restorecon -v /var/named/forwarders.conf
chmod -v 755 /var/named/forwarders.conf
KEYFILE=/var/named/${DOMAIN}.key


rm -rvf /var/named/dynamic
setup DNSSEC key pair
mkdir -vp /var/named/dynamic
cd /var/named/
dnssec-keygen -a HMAC-MD5 -b 512 -n USER -r /dev/urandom ${DOMAIN}
KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)"
cd -
rndc-confgen -a -r /dev/urandom
echo $KEY


echo $domain
setup permissions for the DNSSEC key pair
restorecon -v /etc/rndc.* /etc/named.*
chown -v root:named /etc/rndc.key
chmod -v 640 /etc/rndc.key


cat <<EOF > /var/named/dynamic/${domain}.db
setup forwarders
\$ORIGIN .
echo "forwarders { 8.8.8.8; 8.8.4.4; } ;" >> /var/named/forwarders.conf
\$TTL 1 ; 1 seconds (for testing only)
restorecon -v /var/named/forwarders.conf
${domain} IN SOA ns1.${domain}. hostmaster.${domain}. (
chmod -v 755 /var/named/forwarders.conf
                        2011112904 ; serial
                        60        ; refresh (1 minute)
                        15        ; retry (15 seconds)
                        1800      ; expire (30 minutes)
                        10        ; minimum (10 seconds)
                          )
                    NS ns1.${domain}.
                    MX 10 mail.${domain}.
\$ORIGIN ${domain}.
ns1               A        127.0.0.1


EOF
setup initial DNS database
rm -rvf /var/named/dynamic
mkdir -vp /var/named/dynamic
cat <<EOF > /var/named/dynamic/${DOMAIN}.db
\$ORIGIN .
\$TTL 1 ; 1 seconds (for testing only)
${DOMAIN} IN SOA ns1.${DOMAIN}. hostmaster.${DOMAIN}. (
                          2011112904 ; serial
                          60        ; refresh (1 minute)
                          15        ; retry (15 seconds)
                          1800      ; expire (30 minutes)
                          10        ; minimum (10 seconds)
                          )
                      NS ns1.${DOMAIN}.
                      MX 10 mail.${DOMAIN}.
\$ORIGIN ${DOMAIN}.
ns1               A        127.0.0.1
EOF


cat <<EOF > /var/named/${domain}.key
Install the DNSSEC key
key ${domain} {
cat <<EOF > ${KEYFILE}
  algorithm HMAC-MD5;
key ${DOMAIN} {
  secret "${KEY}";
  algorithm HMAC-MD5;
};
  secret "${KEY}";
EOF
};
EOF


cat /var/named/dynamic/${domain}.db
Check the key and database
cat /var/named/${domain}.key
cat /var/named/dynamic/${DOMAIN}.db
cat /var/named/${DOMAIN}.key


chown -Rv named:named /var/named
Set permissions for key and database
restorecon -rv /var/named
chown -Rv named:named /var/named
restorecon -rv /var/named


mv /etc/named.conf /etc/named.conf.openshift
Create the named configuration file
cat <<EOF > /etc/named.conf
mv /etc/named.conf /etc/named.conf.openshift
// named.conf
cat <<EOF > /etc/named.conf
//
// named.conf
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
//
// server as a caching only nameserver (as a localhost DNS resolver only).
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
//
// server as a caching only nameserver (as a localhost DNS resolver only).
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
 
//
options {
    listen-on port 53 { any; };
options {
    directory "/var/named";
    listen-on port 53 { any; };
    dump-file "/var/named/data/cache_dump.db";
    directory "/var/named";
    statistics-file "/var/named/data/named_stats.txt";
    dump-file "/var/named/data/cache_dump.db";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    statistics-file "/var/named/data/named_stats.txt";
    allow-query { any; };
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    recursion yes;
    allow-query { any; };
 
    recursion yes;
    /* Path to ISC DLV key */
    bindkeys-file "/etc/named.iscdlv.key";
    /* Path to ISC DLV key */
 
    bindkeys-file "/etc/named.iscdlv.key";
    // set forwarding to the next nearest server (from DHCP response
    forward only;
    // set forwarding to the next nearest server (from DHCP response)
    include "forwarders.conf";
    forward only;
};
    include "forwarders.conf";
 
};
logging {
    channel default_debug {
logging {
        file "data/named.run";
    channel default_debug {
        severity dynamic;
        file "data/named.run";
    };
        severity dynamic;
};
    };
 
};
// use the default rndc key
include "/etc/rndc.key";
// use the default rndc key
include "/etc/rndc.key";
 
controls {
    inet 127.0.0.1 port 953
    allow { 127.0.0.1; } keys { "rndc-key"; };
};
include "/etc/named.rfc1912.zones";
include "${DOMAIN}.key";
   
   
controls {
zone "${DOMAIN}" IN {
    inet 127.0.0.1 port 953
    type master;
    allow { 127.0.0.1; } keys { "rndc-key"; };
    file "dynamic/${DOMAIN}.db";
};
    allow-update { key ${DOMAIN} ; } ;
};
EOF


include "/etc/named.rfc1912.zones";
Check the named config file
cat /etc/named.conf


include "${domain}.key";
setup permissions of named config file
chown -v root:named /etc/named.conf
restorecon /etc/named.conf


zone "${domain}" IN {
Setup firewall
    type master;
firewall-cmd --add-service=dns
    file "dynamic/${domain}.db";
firewall-cmd --permanent --add-service=dns
    allow-update { key ${domain} ; } ;
firewall-cmd --list-all
};
EOF


cat /etc/named.conf
Setup and start service
chown -v root:named /etc/named.conf
/bin/systemctl enable named.service
restorecon /etc/named.conf
/bin/systemctl start named.service


vi /etc/resolv.conf
add entries using nsupdate
# Add the following as the first nameserver in the file
nsupdate -k ${KEYFILE}
# nameserver **your broker ip address**
> server 127.0.0.1
> update delete broker.example.com A
> update add **your broker full name ** 180 A **your broker ip address**
(example: update add broker.example.com 180 A 192.168.122.220 )
> send
> quit


firewall-cmd --add-service=dns
Test DNS server
firewall-cmd --permanent --add-service=dns
This is best done before hostname has been set.
firewall-cmd --list-all
ping broker.example.com
/bin/systemctl enable named.service
dig @127.0.0.1 broker.example.com
/bin/systemctl start named.service


nsupdate -k ${keyfile}
== '''Broker: DHCP client and hostname''' ==
> server 127.0.0.1
> update delete broker.example.com A
> update add broker.example.com 180 A **your broker ip address**
> send
> quit


ping broker.example.com
Setup dhcp client
dig @127.0.0.1 broker.example.com
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede host-name \"broker\";" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf


Setup hostname
echo "broker.example.com" > /etc/hostname


== Configure the BROKER DHCP client and hostname ==
== '''Broker: MongoDB''' ==
Install Software
yum -y install mongodb-server


echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
Tweak config file
echo "supersede host-name \"broker\";" >> /etc/dhcp/dhclient-eth0.conf
vi /etc/mongodb.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf
# Uncomment auth = true
# Add smallfiles = true


echo "broker.example.com" > /etc/hostname
Setup and start service
/usr/bin/systemctl enable mongod.service
/usr/bin/systemctl status mongod.service
/usr/bin/systemctl start mongod.service
/usr/bin/systemctl status mongod.service


== Installing and configuring MongoDB ==
Testing
mongo
> show dbs
> exit


yum -y install mongodb-server
== '''Broker: Messaging (using QPID)''' ==
Activemq on F19 isn't ready for OpenShift production.  When it is, we'll use that
For now we'll use QPID with mcollective.


vi /etc/mongodb.conf
Install Software
# Uncomment auth = true
yum install mcollective-qpid-plugin qpid-cpp-server
# Add smallfiles = true


/usr/bin/systemctl enable mongod.service
Setup Firewall
/usr/bin/systemctl status mongod.service
firewall-cmd --add-port=5672/tcp
/usr/bin/systemctl start mongod.service
firewall-cmd --permanent --add-port=5672/tcp
/usr/bin/systemctl status mongod.service
firewall-cmd --list-all


# Testing
Setup and start service
mongo
/usr/bin/systemctl enable qpidd.service
> show dbs
/usr/bin/systemctl start qpidd.service
> exit
/usr/bin/systemctl status qpidd.service


== Installing and configuring QPID ==
== '''Broker: MCollective client ( using QPID)''' ==
# Activemq on F19 isn't ready for production. When it is, we'll use that
Install Software
# For now let's use QPID with mcollective.
  yum -y install mcollective-client


yum install mcollective-qpid-plugin qpid-cpp-server
Move original config file out of the way
firewall-cmd --add-port=5672/tcp
mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig
firewall-cmd --permanent --add-port=5672/tcp
firewall-cmd --list-all


/usr/bin/systemctl enable qpidd.service
Create new client config file.  This config file is for using QPID as a messaging platform.
/usr/bin/systemctl start qpidd.service
cat <<EOF > /etc/mcollective/client.cfg
/usr/bin/systemctl status qpidd.service
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
loglevel = debug
logfile = /var/log/mcollective-client.log
# Plugins
securityprovider = psk
plugin.psk = unset
connector = qpid
plugin.qpid.host=${BROKERNAME}
plugin.qpid.secure=false
plugin.qpid.timeout=5
# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
EOF


== Installing and configuring MCollective client (QPID) ==
== '''Broker: broker application''' ==
yum -y install mcollective-client
Install software
mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig
yum -y install openshift-origin-broker openshift-origin-broker-util rubygem-openshift-origin-auth-remote-user rubygem-openshift-origin-msg-broker-mcollective rubygem-openshift-origin-dns-bind


cat <<EOF > /etc/mcollective/client.cfg
Modify the broker proxy server name
topicprefix = /topic/
sed -i -e "s/ServerName .*$/ServerName broker.example.com/" /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf
main_collective = mcollective
cat /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf
collectives = mcollective
libdir = /usr/libexec/mcollective
loglevel = debug
logfile = /var/log/mcollective-client.log


# Plugins
Setup and start service
securityprovider = psk
/usr/bin/systemctl enable httpd.service
plugin.psk = unset
/usr/bin/systemctl enable ntpd.service
connector = qpid
/usr/bin/systemctl enable sshd.service
plugin.qpid.host=broker.example.com
plugin.qpid.secure=false
plugin.qpid.timeout=5


# Facts
Setup Firewall
factsource = yaml
firewall-cmd --add-service=ssh
plugin.yaml = /etc/mcollective/facts.yaml
firewall-cmd --add-service=http
EOF
firewall-cmd --add-service=https
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all


== Installing and configuring the broker application ==
Generate access key
# When mcollective was updated to 2.2.3 it created a conflict with one of our components.
openssl genrsa -out /etc/openshift/server_priv.pem 2048
# We are working on fixing the conflict, but until then, do the following.
openssl rsa -in /etc/openshift/server_priv.pem -pubout > /etc/openshift/server_pub.pem
yumdownloader openshift-origin-msg-common
ssh-keygen -t rsa -b 2048 -f ~/.ssh/rsync_id_rsa
rpm -Uvh openshift-origin-msg-common-1.4.1-1.fc19.noarch.rpm --nodeps --force
cp -v ~/.ssh/rsync_id_rsa* /etc/openshift/


yum -y install openshift-origin-broker openshift-origin-broker-util rubygem-openshift-origin-auth-remote-user rubygem-openshift-origin-msg-broker-mcollective rubygem-openshift-origin-dns-bind
Setup selinux boolean variables and set file contexts
setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_run_stickshift=on named_write_master_zones=on
fixfiles -R rubygem-passenger restore
fixfiles -R mod_passenger restore
restorecon -rv /var/run
restorecon -rv /usr/share/gems/gems/passenger-*


sed -i -e "s/ServerName .*$/ServerName broker.example.com/" /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf
Tweak broker config, if needed
cat /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf
vi /etc/openshift/broker.conf
# Might not have to do anything but make sure you have the following lines
CLOUD_DOMAIN="example.com"
VALID_GEAR_SIZES="small,medium"


/usr/bin/systemctl enable httpd.service
== '''Broker: broker plugins and MongoDB user accounts''' ==
/usr/bin/systemctl enable ntpd.service
Create config files from examples
/usr/bin/systemctl enable sshd.service
cp /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf.example /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf


firewall-cmd --add-service=ssh
Config the DNS plugin
firewall-cmd --add-service=http
cd /var/named/
firewall-cmd --add-service=https
KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)"
firewall-cmd --permanent --add-service=ssh
cat $KEYFILE
firewall-cmd --permanent --add-service=http
echo $KEY
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all
cat <<EOF > /etc/openshift/plugins.d/openshift-origin-dns-bind.conf
BIND_SERVER="127.0.0.1"
BIND_PORT=53
BIND_KEYNAME="${DOMAIN}"
BIND_KEYVALUE="${KEY}"
BIND_ZONE="${DOMAIN}"
EOF


openssl genrsa -out /etc/openshift/server_priv.pem 2048
Configure authentication plugin and add a user
openssl rsa -in /etc/openshift/server_priv.pem -pubout > /etc/openshift/server_pub.pem
cp -v /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user-basic.conf.sample /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user.conf
ssh-keygen -t rsa -b 2048 -f ~/.ssh/rsync_id_rsa
htpasswd -c -b -s /etc/openshift/htpasswd demo demo
cp -v ~/.ssh/rsync_id_rsa* /etc/openshift/
# Don't forget your password. <demo password>
cat /etc/openshift/htpasswd


setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_run_stickshift=on named_write_master_zones=on
Add MongoDB account
fixfiles -R rubygem-passenger restore
grep MONGO /etc/openshift/broker.conf
fixfiles -R mod_passenger restore
mongo openshift_broker_dev --eval 'db.addUser("openshift", "mooo")'
restorecon -rv /var/run
# If you are going to change the username and/or password, change broker.conf
restorecon -rv /usr/share/gems/gems/passenger-*


vi /etc/openshift/broker.conf
Bundle broker gems
# Might not have to do anything
yum -y install rubygem-psych rubygem-mocha
CLOUD_DOMAIN="example.com"
cd /var/www/openshift/broker
VALID_GEAR_SIZES="small,medium"
gem install mongoid
bundle --local


== Configuring the broker plugins and MongoDB user accounts ==
Setup and start services
/usr/bin/systemctl enable openshift-broker.service
/usr/bin/systemctl start httpd.service
/usr/bin/systemctl start openshift-broker.service
/usr/bin/systemctl status openshift-broker.service


cp /usr/share/gems/gems/openshift-origin-auth-remote-user-*/conf/openshift-origin-auth-remote-user.conf.example /etc/openshift/plugins.d/openshift-origin-auth-remote-user.conf
Test basic broker service
cp /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf.example /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf
curl -k -u demo:demo https://localhost/broker/rest/api


domain=example.com
= '''''Setup and Configure Node''''' =
keyfile=/var/named/${domain}.key
cd /var/named/
KEY="$(grep Key: K${domain}*.private | cut -d ' ' -f 2)"
cat $keyfile
echo $KEY


cd /etc/openshift/plugins.d/
== '''Node: Initial setup/configure''' ==
cat <<EOF > openshift-origin-dns-bind.conf
'''ON BROKER'''
BIND_SERVER="127.0.0.1"
KEYFILE=/var/named/${DOMAIN}.key
BIND_PORT=53
BIND_KEYNAME="${domain}"
BIND_KEYVALUE="${KEY}"
BIND_ZONE="${domain}"
EOF


#pushd /usr/share/selinux/packages/rubygem-openshift-origin-dns-bind/ && make -f /usr/share/selinux/devel/Makefile ; popd
Register the node in DNS
#semodule -i /usr/share/selinux/packages/rubygem-openshift-origin-dns-bind/dhcpnamedforward.pp
oo-register-dns -h ${NODENAME} -d ${DOMAIN} -n ${NODEIP} -k ${KEYFILE}


cp -v /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user-basic.conf.sample /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user.conf
Copy the broker public key to node
htpasswd -c -b -s /etc/openshift/htpasswd demo demopassword
scp /etc/openshift/rsync_id_rsa.pub root@${NODENAME}:/root/.ssh/
# Don't forget your password. <demo password>
cat /etc/openshift/htpasswd


grep MONGO /etc/openshift/broker.conf
'''ON NODE'''
mongo openshift_broker_dev --eval 'db.addUser("openshift", "mooo")'
Put the brokers public key in root authorized keys
# If you are going to change the username and/or password, change broker.conf
cat /root/.ssh/rsync_id_rsa.pub >> /root/.ssh/authorized_keys
rm -f /root/.ssh/rsync_id_rsa.pub


yum -y install rubygem-psych
'''ON BROKER'''
cd /var/www/openshift/broker
Test to make sure we can login using our key
# This is being fixed, but for now do the following
ssh -i /root/.ssh/rsync_id_rsa root@${NODENAME}
vi Gemfile
exit
# remove minitest version
# add gem 'psych'
gem install mongoid
bundle --local


/usr/bin/systemctl enable openshift-broker.service
== '''Node: DHCP client and hostname''' ==
/usr/bin/systemctl start httpd.service
Configure the dhcp settings
/usr/bin/systemctl start openshift-broker.service
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
/usr/bin/systemctl status openshift-broker.service
echo "supersede host-name \"node\";" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf


curl -k -u demo:demopassword https://localhost/broker/rest/api
Set the hostname
echo "node.example.com" > /etc/hostname


== '''Node: MCollective'''  ==
'''ON NODE'''
Install Software
yum -y install openshift-origin-msg-node-mcollective mcollective-qpid-plugin


= Setup and Configure Node =
Move original configuration out of the way
== Initial setup/configure of the node host ==
mv /etc/mcollective/server.cfg /etc/mcollective/server.cfg.orig
# ON NODE
yum update
yum -y install ntp
/bin/systemctl enable ntpd.service
/bin/systemctl start  ntpd.service


# Find out the node ip address
Create new configuration
nm-tool
cat <<EOF > /etc/mcollective/server.cfg
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logfile = /var/log/mcollective.log
loglevel = debug
daemonize = 1
direct_addressing = n
# Plugins
securityprovider = psk
plugin.psk = unset
connector = qpid
plugin.qpid.host=${BROKERNAME}
plugin.qpid.secure=false
plugin.qpid.timeout=5
# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
EOF


# ON BROKER
Setup and start services
domain=example.com
/bin/systemctl enable mcollective.service
keyfile=/var/named/${domain}.key
/bin/systemctl start  mcollective.service
# Use the IP address from the node, found above
oo-register-dns -h node -d ${domain} -n 192.168.122.161 -k ${keyfile}


scp /etc/openshift/rsync_id_rsa.pub root@node.example.com:/root/.ssh/
'''ON BROKER'''
mco ping
# node should show up on mco ping


# ON NODE
== '''Node: node application''' ==
cat /root/.ssh/rsync_id_rsa.pub >> /root/.ssh/authorized_keys
Install software
rm -f /root/.ssh/rsync_id_rsa.pub
yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util
yum -y install openshift-origin-cartridge-cron-1.4 openshift-origin-cartridge-diy-0.1


# ON BROKER
Setup firewall
ssh -i /root/.ssh/rsync_id_rsa root@node.example.com
firewall-cmd --add-service=ssh
exit
firewall-cmd --add-service=http
firewall-cmd --add-service=https
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all


#Find out the broker ip address
== '''Node: PAM namespace module, cgroups, and user quotas''' ==
nm-tool
PAM
sed -i -e 's|pam_selinux|pam_openshift|g' /etc/pam.d/sshd
for f in "runuser" "runuser-l" "sshd" "su" "system-auth-ac"
do
  t="/etc/pam.d/$f"
  if ! grep -q "pam_namespace.so" "$t"
  then
    echo -e "session\t\trequired\tpam_namespace.so no_unmount_on_close" >> "$t"
  fi
done


=== Configure the NODE DHCP client and hostname ===
CGROUPS
# ON NODE
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede host-name \"node\";" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf
echo "node.example.com" > /etc/hostname


reboot
Cgroups Config - Need to still fixup the cgroup configurations
echo "mount {" >> /etc/cgconfig.conf
echo "        cpu    = /cgroup/all;" >> /etc/cgconfig.conf
echo "        cpuacct = /cgroup/all;" >> /etc/cgconfig.conf
echo "        memory  = /cgroup/all;" >> /etc/cgconfig.conf
echo "        freezer = /cgroup/all;" >> /etc/cgconfig.conf
echo "        net_cls = /cgroup/all;" >> /etc/cgconfig.conf
echo "}" >> /etc/cgconfig.conf
restorecon -v /etc/cgconfig.conf
mkdir /cgroup
restorecon -RFvv /cgroup


== Setting up MCollective on the node host ==
Cgroups enable and startup services
# ON NODE
/bin/systemctl enable cgconfig.service
yum -y install openshift-origin-msg-node-mcollective
/bin/systemctl enable cgred.service
mv /etc/mcollective/server.cfg /etc/mcollective/server.cfg.orig
/usr/sbin/chkconfig openshift-cgroups on
/bin/systemctl restart  cgconfig.service
/bin/systemctl restart  cgred.service
/usr/sbin/service openshift-cgroups restart


cat <<EOF > /etc/mcollective/server.cfg
DISK QUOTA
topicprefix = /topic/
# Edit fstab and add usrquota to whichever filesystem
main_collective = mcollective
#  has /var/lib/openshift on it
collectives = mcollective
UUID=b9e21eae-4b8c-4936-9f5d-d10631ff535e / ext4    defaults,usrquota 1 1
libdir = /usr/libexec/mcollective
# reboot or remount
logfile = /var/log/mcollective.log
mount -o remount /
loglevel = debug
quotacheck -cmug /
daemonize = 1
direct_addressing = n


# Plugins
== '''Node: SELinux and System Control''' ==
securityprovider = psk
Setup SELINUX Booleans
plugin.psk = unset
setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_read_user_content=on httpd_enable_homedirs=on httpd_run_stickshift=on allow_polyinstantiation=on
connector = qpid
plugin.qpid.host=broker.example.com
plugin.qpid.secure=false
plugin.qpid.timeout=5


# Facts
Update selinux file setting
factsource = yaml
restorecon -rv /var/run
plugin.yaml = /etc/mcollective/facts.yaml
restorecon -rv /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid
EOF
restorecon -rv /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift


/bin/systemctl enable mcollective.service
SYSTEM CONTROL SETTINGS
/bin/systemctl start mcollective.service
echo "# Added for OpenShift" >> /etc/sysctl.d/openshift.conf
echo "kernel.sem = 250  32000 32  4096" >> /etc/sysctl.d/openshift.conf
echo "net.ipv4.ip_local_port_range = 15000 35530" >> /etc/sysctl.d/openshift.conf
echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/openshift.conf
  sysctl -p /etc/sysctl.d/openshift.conf


# ON BROKER
== '''Node: SSH, Port Proxy, and Node application''' ==
mco ping
SSH
vi /etc/ssh/sshd_config
> AcceptEnv GIT_SSH
perl -p -i -e "s/^#MaxSessions .*$/MaxSessions 40/" /etc/ssh/sshd_config
perl -p -i -e "s/^#MaxStartups .*$/MaxStartups 40/" /etc/ssh/sshd_config
/bin/systemctl restart  sshd.service


== Setting up node packages on the node host ==
PORT PROXY
# ON NODE
firewall-cmd --add-port=35531-65535/tcp
yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util
firewall-cmd --permanent --add-port=35531-65535/tcp
yum -y install openshift-origin-cartridge-cron-1.4 openshift-origin-cartridge-diy-0.1
firewall-cmd --list-all
/bin/systemctl enable openshift-port-proxy.service
/bin/systemctl restart  openshift-port-proxy.service


firewall-cmd --add-service=ssh
NODE SETUP
firewall-cmd --add-service=http
/bin/systemctl enable httpd.service
firewall-cmd --add-service=https
/bin/systemctl enable openshift-gears.service
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
vi /etc/openshift/node.conf
firewall-cmd --permanent --add-service=https
> PUBLIC_HOSTNAME="node.example.com"
firewall-cmd --list-all
> PUBLIC_IP="192.168.122.161" (Node IP Address)
> BROKER_HOST="192.168.122.220" (Broker IP Address)
> CLOUD_DOMAIN="example.com"
/etc/cron.minutely/openshift-facts


== Configuring PAM namespace module, cgropus, and user quotas on the node host ==
== '''Node: Reboot''' ==
# ON NODE
We need to reboot to load all the node stuff correctly
# PAM
reboot
sed -i -e 's|pam_selinux|pam_openshift|g' /etc/pam.d/sshd


for f in "runuser" "runuser-l" "sshd" "su" "system-auth-ac"
= '''''Testing''''' =
do
==Test on Broker (after node is back up)==
  t="/etc/pam.d/$f"
  if ! grep -q "pam_namespace.so" "$t"
  then
    echo -e "session\t\trequired\tpam_namespace.so no_unmount_on_close" >> "$t"
  fi
done


#CGROUPS
'''Check Messaging'''
#echo "mount {" >> /etc/cgconfig.conf
mco ping
#echo "        cpu    = /cgroup/all;" >> /etc/cgconfig.conf
Should look like
#echo "        cpuacct = /cgroup/all;" >> /etc/cgconfig.conf
node.example.com                        time=239.51 ms
#echo "        memory = /cgroup/all;" >> /etc/cgconfig.conf
   
#echo "        freezer = /cgroup/all;" >> /etc/cgconfig.conf
---- ping statistics ----
#echo "        net_cls = /cgroup/all;" >> /etc/cgconfig.conf
1 replies max: 239.51 min: 239.51 avg: 239.51
#echo "}" >> /etc/cgconfig.conf
#restorecon -v /etc/cgconfig.conf
#mkdir /cgroup
#restorecon -RFvv /cgroup


/bin/systemctl enable cgconfig.service
'''Check Broker'''
/bin/systemctl enable cgred.service
curl -k -u demo:demo https://localhost/broker/rest/api
/usr/sbin/chkconfig openshift-cgroups on
Should look like
/bin/systemctl restart cgconfig.service
{"data":{"API":{"href":"https://localhost/broker/rest/api","method":"GET","optional_params":[],"rel":"API entry point","required_params":[]},"GET_ENVIRONMENT":{"href":"https://localhost/broker/rest/environment","method":"GET","optional_params":[],"rel":"Get environment information","required_params":[]},"GET_USER"
/bin/systemctl restart cgred.service
  ...
/usr/sbin/service openshift-cgroups restart
  :id","type":"string","valid_options":[]}]}},"messages":[],"status":"ok","supported_api_versions":[1.0,1.1,1.2,1.3],"type":"links","version":"1.3"}


#DISK QUOTA
'''Check and Setup User'''
# Edit fstab and add usrquota to whichever filesystem
yum -y install rubygem-rhc
#   has /var/lib/openshift on it
LIBRA_SERVER=broker.example.com rhc setup
UUID=b9e21eae-4b8c-4936-9f5d-d10631ff535e / ext4    defaults,usrquota 1 1
Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot)
# reboot or remount
OpenShift Client Tools (RHC) Setup Wizard
mount -o remount /
quotacheck -cmug /
This wizard will help you upload your SSH keys, set your application namespace, and
check that other programs like Git are properly installed.
The server's certificate is self-signed, which means that a secure connection can't be
established to 'broker.example.com'.
You may bypass this check, but any data you send to the server could be intercepted by
others.
Connect without checking the certificate? (yes|no): yes
Login to broker.example.com: demo
Password: ****
    
OpenShift can create and store a token on disk which allows to you to access the
server without using your password. The key is stored in your home directory and
should be kept secret.  You can delete the key at any time by running 'rhc logout'.
Generate a token now? (yes|no) no
Saving configuration to /root/.openshift/express.conf ... done
No SSH keys were found. We will generate a pair of keys for you.
    Created: /root/.ssh/id_rsa.pub
Your public SSH key must be uploaded to the OpenShift server to access code.  Upload
now? (yes|no)
yes
Since you do not have any keys associated with your OpenShift account, your new key
will be uploaded as the 'default' key.
Uploading key 'default' ... done
Checking for git ... found git version 1.8.2.1
Checking common problems .. done
Checking your namespace ... none
Your namespace is unique to your account and is the suffix of the public URLs we
assign to your applications. You may configure your namespace here or leave it blank
and use 'rhc create-domain' to create a namespace later.  You will not be able to
create applications without first creating a namespace.
Please enter a namespace (letters and numbers only) |<none>|: demoland


== Configuring SELinux and System Control on the node host ==
Create an app
# ON NODE
rhc domain show -p demo
# SELINUX
rhc app create test1 diy-0.1 -p demo
setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_read_user_content=on httpd_enable_homedirs=on httpd_run_stickshift=on allow_polyinstantiation=on


restorecon -rv /var/run
==Test on Local Machine (after node is back up)==
restorecon -rv /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid
Setup your machine to use broker as a name server (Note: This might mess up normal network operations.)
restorecon -rv /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift
vi /etc/resolve.conf
# At the first line put "nameserver *broker ip address*"
nameserver 192.168.122.220


# SYSTEM CONTROL SETTINGS
'''Check and Setup User'''
echo "# Added for OpenShift" >> /etc/sysctl.d/openshift.conf
yum -y install rubygem-rhc
echo "kernel.sem = 250 32000 32 4096" >> /etc/sysctl.d/openshift.conf
LIBRA_SERVER=broker.example.com rhc setup
echo "net.ipv4.ip_local_port_range = 15000 35530" >> /etc/sysctl.d/openshift.conf
Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot)
echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/openshift.conf
OpenShift Client Tools (RHC) Setup Wizard
sysctl -p /etc/sysctl.d/openshift.conf
 
This wizard will help you upload your SSH keys, set your application namespace, and
== Configuring SSH, Port Proxy, and Node on the node host ==
check that other programs like Git are properly installed.
# ON NODE
# SSH
  The server's certificate is self-signed, which means that a secure connection can't be
vi /etc/ssh/sshd_config
  established to 'broker.example.com'.
> AcceptEnv GIT_SSH
 
You may bypass this check, but any data you send to the server could be intercepted by
perl -p -i -e "s/^#MaxSessions .*$/MaxSessions 40/" /etc/ssh/sshd_config
others.
perl -p -i -e "s/^#MaxStartups .*$/MaxStartups 40/" /etc/ssh/sshd_config
 
Connect without checking the certificate? (yes|no): yes
/bin/systemctl restart sshd.service
Login to broker.example.com: demo
 
Password: ****
# PORT PROXY
 
 
OpenShift can create and store a token on disk which allows to you to access the
firewall-cmd --add-port=35531-65535/tcp
server without using your password. The key is stored in your home directory and
firewall-cmd --permanent --add-port=35531-65535/tcp
should be kept secret. You can delete the key at any time by running 'rhc logout'.
firewall-cmd --list-all
Generate a token now? (yes|no) no
 
/bin/systemctl enable openshift-port-proxy.service
Saving configuration to /root/.openshift/express.conf ... done
/bin/systemctl restart openshift-port-proxy.service
 
No SSH keys were found. We will generate a pair of keys for you.
# NODE SETUP
/bin/systemctl enable openshift-gears.service
    Created: /root/.ssh/id_rsa.pub
 
# Find node and broker IP address
Your public SSH key must be uploaded to the OpenShift server to access code. Upload
nm-tool
now? (yes|no)
 
yes
vi /etc/openshift/node.conf
   
> PUBLIC_HOSTNAME="node.example.com"
Since you do not have any keys associated with your OpenShift account, your new key
> PUBLIC_IP="192.168.122.161" (Node IP Address)
will be uploaded as the 'default' key.
> BROKER_HOST="192.168.122.220" (Broker IP Address)
> CLOUD_DOMAIN="example.com"
Uploading key 'default' ... done
Checking for git ... found git version 1.8.2.1
Checking common problems .. done
  Checking your namespace ... none
Your namespace is unique to your account and is the suffix of the public URLs we
assign to your applications. You may configure your namespace here or leave it blank
and use 'rhc create-domain' to create a namespace later. You will not be able to
create applications without first creating a namespace.
Please enter a namespace (letters and numbers only) |<none>|: demoland


/etc/cron.minutely/openshift-facts
Create an app
rhc domain show -p demo
rhc app create test2 diy-0.1 -p demo


== Reboot Node and test ==
# ON NODE
reboot


# ON BROKER (after node is back up)
'''Check App'''
mco ping
You should be able to go to the following URL in your web browser.
curl -k -u demo:demo https://localhost/broker/rest/api


yum -y install rubygem-rhc
http://test2-demoland.example.com/
LIBRA_SERVER=broker.example.com rhc setup

Latest revision as of 13:39, 8 August 2013

Fedora 19 is when OpenShift Origin first became a feature.

NOTE: (August 8, 2013) This page is getting an update. It will accommodate F19 cloud images (not just minimal install). It is also updated with the OpenShift Origin Version 2 documentation.

This page is here to show how to setup OpenShift Origin on Fedora 19 using the packages in Fedora, as opposed to the packages published from upstream. These steps are written out to be done by hand. Yes, people can script and/or puppetize these steps. But these are written out so that people can see, and fine tune them.

Note: And now they have been written into scripts. https://github.com/tdawson/oo-install-scripts

Goal: By the end of this, you should have two machines. A broker machine, and one node machine. You should be able to create applications, that will be put on the node machine. You should be able to check the status of those applications. You should be able to point your web browser to the URL of those applications.

Note: There is no web console in Fedora 19. That will be in Fedora 20.

These instructions were created most from the following two places.

Initial Setup of Broker and Node Machines

ON BOTH BROKER AND NODE 

# Start with a Fedora 19 minimal install
yum -y update
# avoid clock skew
yum -y install ntp
/bin/systemctl enable ntpd.service
/bin/systemctl start  ntpd.service

ON BROKER 

export DOMAIN="example.com"
export BROKERIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"
export BROKERNAME="broker.example.com"
export NODEIP="--- IP Address from Node machine ---"
export NODENAME="node.example.com"
# Here is the IP Address from Broker machine
nm-tool | grep Address | grep -v HW | awk '{print $2}'

ON NODE 

export DOMAIN="example.com"
export BROKERIP="--- IP Address from Broker machine ---"
export BROKERNAME="broker.example.com"
export NODEIP="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"
export NODENAME="node.example.com"
# Here is the IP Address from Node machine
nm-tool | grep Address | grep -v HW | awk '{print $2}'

Setup and Configure Broker

Broker: Bind DNS

yum -y install bind bind-utils

KEYFILE=/var/named/${DOMAIN}.key

setup DNSSEC key pair 

cd /var/named/
dnssec-keygen -a HMAC-MD5 -b 512 -n USER -r /dev/urandom ${DOMAIN}
KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)"
cd -
rndc-confgen -a -r /dev/urandom
echo $KEY

setup permissions for the DNSSEC key pair 

restorecon -v /etc/rndc.* /etc/named.*
chown -v root:named /etc/rndc.key
chmod -v 640 /etc/rndc.key

setup forwarders 

echo "forwarders { 8.8.8.8; 8.8.4.4; } ;" >> /var/named/forwarders.conf
restorecon -v /var/named/forwarders.conf
chmod -v 755 /var/named/forwarders.conf

setup initial DNS database 

rm -rvf /var/named/dynamic
mkdir -vp /var/named/dynamic

cat <<EOF > /var/named/dynamic/${DOMAIN}.db
\$ORIGIN .
\$TTL 1	; 1 seconds (for testing only)
${DOMAIN} IN SOA ns1.${DOMAIN}. hostmaster.${DOMAIN}. (
                         2011112904 ; serial
                         60         ; refresh (1 minute)
                         15         ; retry (15 seconds)
                         1800       ; expire (30 minutes)
                         10         ; minimum (10 seconds)
                          )
                     NS ns1.${DOMAIN}.
                     MX 10 mail.${DOMAIN}.
\$ORIGIN ${DOMAIN}.
ns1	              A        127.0.0.1

EOF

Install the DNSSEC key 

cat <<EOF > ${KEYFILE}
key ${DOMAIN} {
  algorithm HMAC-MD5;
  secret "${KEY}";
};
EOF

Check the key and database 

cat /var/named/dynamic/${DOMAIN}.db
cat /var/named/${DOMAIN}.key

Set permissions for key and database 

chown -Rv named:named /var/named
restorecon -rv /var/named

Create the named configuration file 

mv /etc/named.conf /etc/named.conf.openshift
cat <<EOF > /etc/named.conf
// named.conf
//
// Provided by Red Hat bind package to configure the ISC BIND named(8) DNS
// server as a caching only nameserver (as a localhost DNS resolver only).
//
// See /usr/share/doc/bind*/sample/ for example named configuration files.
//

options {
    listen-on port 53 { any; };
    directory "/var/named";
    dump-file "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    allow-query { any; };
    recursion yes;

    /* Path to ISC DLV key */
    bindkeys-file "/etc/named.iscdlv.key";

    // set forwarding to the next nearest server (from DHCP response)
    forward only;
    include "forwarders.conf";
};

logging {
    channel default_debug {
        file "data/named.run";
        severity dynamic;
    };
};

// use the default rndc key
include "/etc/rndc.key";
 
controls {
    inet 127.0.0.1 port 953
    allow { 127.0.0.1; } keys { "rndc-key"; };
};

include "/etc/named.rfc1912.zones";

include "${DOMAIN}.key";

zone "${DOMAIN}" IN {
    type master;
    file "dynamic/${DOMAIN}.db";
    allow-update { key ${DOMAIN} ; } ;
};
EOF

Check the named config file 

cat /etc/named.conf

setup permissions of named config file 

chown -v root:named /etc/named.conf
restorecon /etc/named.conf

Setup firewall 

firewall-cmd --add-service=dns
firewall-cmd --permanent --add-service=dns
firewall-cmd --list-all

Setup and start service 

/bin/systemctl enable named.service
/bin/systemctl start named.service

add entries using nsupdate 

nsupdate -k ${KEYFILE}
> server 127.0.0.1
> update delete broker.example.com A
> update add **your broker full name ** 180 A **your broker ip address**
(example: update add broker.example.com 180 A 192.168.122.220 )
> send
> quit

Test DNS server This is best done before hostname has been set. 

ping broker.example.com
dig @127.0.0.1 broker.example.com

Broker: DHCP client and hostname

Setup dhcp client 

echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede host-name \"broker\";" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf

Setup hostname 

echo "broker.example.com" > /etc/hostname

Broker: MongoDB

Install Software 

yum -y install mongodb-server

Tweak config file 

vi /etc/mongodb.conf
# Uncomment auth = true
# Add smallfiles = true

Setup and start service 

/usr/bin/systemctl enable mongod.service
/usr/bin/systemctl status mongod.service
/usr/bin/systemctl start mongod.service
/usr/bin/systemctl status mongod.service

Testing 

mongo
> show dbs
> exit

Broker: Messaging (using QPID)

Activemq on F19 isn't ready for OpenShift production. When it is, we'll use that For now we'll use QPID with mcollective.

Install Software 

yum install mcollective-qpid-plugin qpid-cpp-server

Setup Firewall 

firewall-cmd --add-port=5672/tcp
firewall-cmd --permanent --add-port=5672/tcp
firewall-cmd --list-all

Setup and start service 

/usr/bin/systemctl enable qpidd.service
/usr/bin/systemctl start qpidd.service
/usr/bin/systemctl status qpidd.service

Broker: MCollective client ( using QPID)

Install Software 

yum -y install mcollective-client

Move original config file out of the way 

mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig

Create new client config file. This config file is for using QPID as a messaging platform. 

cat <<EOF > /etc/mcollective/client.cfg
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
loglevel = debug
logfile = /var/log/mcollective-client.log

# Plugins
securityprovider = psk
plugin.psk = unset
connector = qpid
plugin.qpid.host=${BROKERNAME}
plugin.qpid.secure=false
plugin.qpid.timeout=5

# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
EOF

Broker: broker application

Install software 

yum -y install openshift-origin-broker openshift-origin-broker-util rubygem-openshift-origin-auth-remote-user rubygem-openshift-origin-msg-broker-mcollective rubygem-openshift-origin-dns-bind

Modify the broker proxy server name 

sed -i -e "s/ServerName .*$/ServerName broker.example.com/" /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf 
cat /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf

Setup and start service 

/usr/bin/systemctl enable httpd.service
/usr/bin/systemctl enable ntpd.service
/usr/bin/systemctl enable sshd.service

Setup Firewall 

firewall-cmd --add-service=ssh
firewall-cmd --add-service=http
firewall-cmd --add-service=https
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all

Generate access key 

openssl genrsa -out /etc/openshift/server_priv.pem 2048
openssl rsa -in /etc/openshift/server_priv.pem -pubout > /etc/openshift/server_pub.pem
ssh-keygen -t rsa -b 2048 -f ~/.ssh/rsync_id_rsa
cp -v ~/.ssh/rsync_id_rsa* /etc/openshift/

Setup selinux boolean variables and set file contexts 

setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_run_stickshift=on named_write_master_zones=on
fixfiles -R rubygem-passenger restore
fixfiles -R mod_passenger restore
restorecon -rv /var/run
restorecon -rv /usr/share/gems/gems/passenger-*

Tweak broker config, if needed 

vi /etc/openshift/broker.conf
# Might not have to do anything but make sure you have the following lines
CLOUD_DOMAIN="example.com"
VALID_GEAR_SIZES="small,medium"

Broker: broker plugins and MongoDB user accounts

Create config files from examples 

cp /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf.example /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf

Config the DNS plugin 

cd /var/named/
KEY="$(grep Key: K${DOMAIN}*.private | cut -d ' ' -f 2)"
cat $KEYFILE
echo $KEY

cat <<EOF > /etc/openshift/plugins.d/openshift-origin-dns-bind.conf
BIND_SERVER="127.0.0.1"
BIND_PORT=53
BIND_KEYNAME="${DOMAIN}"
BIND_KEYVALUE="${KEY}"
BIND_ZONE="${DOMAIN}"
EOF

Configure authentication plugin and add a user 

cp -v /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user-basic.conf.sample /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user.conf
htpasswd -c -b -s /etc/openshift/htpasswd demo demo
# Don't forget your password. <demo password>
cat /etc/openshift/htpasswd

Add MongoDB account 

grep MONGO /etc/openshift/broker.conf
mongo openshift_broker_dev --eval 'db.addUser("openshift", "mooo")'
# If you are going to change the username and/or password, change broker.conf

Bundle broker gems 

yum -y install rubygem-psych rubygem-mocha
cd /var/www/openshift/broker
gem install mongoid
bundle --local

Setup and start services 

/usr/bin/systemctl enable openshift-broker.service
/usr/bin/systemctl start httpd.service
/usr/bin/systemctl start openshift-broker.service
/usr/bin/systemctl status openshift-broker.service

Test basic broker service 

curl -k -u demo:demo https://localhost/broker/rest/api

Setup and Configure Node

Node: Initial setup/configure

ON BROKER 

KEYFILE=/var/named/${DOMAIN}.key

Register the node in DNS 

oo-register-dns -h ${NODENAME} -d ${DOMAIN} -n ${NODEIP} -k ${KEYFILE}

Copy the broker public key to node 

scp /etc/openshift/rsync_id_rsa.pub root@${NODENAME}:/root/.ssh/

ON NODE Put the brokers public key in root authorized keys 

cat /root/.ssh/rsync_id_rsa.pub >> /root/.ssh/authorized_keys
rm -f /root/.ssh/rsync_id_rsa.pub

ON BROKER Test to make sure we can login using our key 

ssh -i /root/.ssh/rsync_id_rsa root@${NODENAME}
exit

Node: DHCP client and hostname

Configure the dhcp settings 

echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede host-name \"node\";" >> /etc/dhcp/dhclient-eth0.conf
echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf

Set the hostname 

echo "node.example.com" > /etc/hostname

Node: MCollective

ON NODE Install Software 

yum -y install openshift-origin-msg-node-mcollective mcollective-qpid-plugin

Move original configuration out of the way 

mv /etc/mcollective/server.cfg /etc/mcollective/server.cfg.orig

Create new configuration 

cat <<EOF > /etc/mcollective/server.cfg
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/libexec/mcollective
logfile = /var/log/mcollective.log
loglevel = debug
daemonize = 1
direct_addressing = n

# Plugins
securityprovider = psk
plugin.psk = unset
connector = qpid
plugin.qpid.host=${BROKERNAME}
plugin.qpid.secure=false
plugin.qpid.timeout=5

# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml
EOF

Setup and start services 

/bin/systemctl enable mcollective.service
/bin/systemctl start  mcollective.service

ON BROKER 

mco ping
# node should show up on mco ping

Node: node application

Install software 

yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util
yum -y install openshift-origin-cartridge-cron-1.4 openshift-origin-cartridge-diy-0.1

Setup firewall 

firewall-cmd --add-service=ssh
firewall-cmd --add-service=http
firewall-cmd --add-service=https
firewall-cmd --permanent --add-service=ssh
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --list-all

Node: PAM namespace module, cgroups, and user quotas

PAM 

sed -i -e 's|pam_selinux|pam_openshift|g' /etc/pam.d/sshd

for f in "runuser" "runuser-l" "sshd" "su" "system-auth-ac"
do
  t="/etc/pam.d/$f"
  if ! grep -q "pam_namespace.so" "$t"
  then
    echo -e "session\t\trequired\tpam_namespace.so no_unmount_on_close" >> "$t"
  fi
done

CGROUPS

Cgroups Config - Need to still fixup the cgroup configurations 

echo "mount {" >> /etc/cgconfig.conf
echo "        cpu     = /cgroup/all;" >> /etc/cgconfig.conf
echo "        cpuacct = /cgroup/all;" >> /etc/cgconfig.conf
echo "        memory  = /cgroup/all;" >> /etc/cgconfig.conf
echo "        freezer = /cgroup/all;" >> /etc/cgconfig.conf
echo "        net_cls = /cgroup/all;" >> /etc/cgconfig.conf
echo "}" >> /etc/cgconfig.conf
restorecon -v /etc/cgconfig.conf
mkdir /cgroup
restorecon -RFvv /cgroup

Cgroups enable and startup services 

/bin/systemctl enable cgconfig.service
/bin/systemctl enable cgred.service
/usr/sbin/chkconfig openshift-cgroups on
/bin/systemctl restart  cgconfig.service
/bin/systemctl restart  cgred.service
/usr/sbin/service openshift-cgroups restart

DISK QUOTA 

# Edit fstab and add usrquota to whichever filesystem 
#   has /var/lib/openshift on it
UUID=b9e21eae-4b8c-4936-9f5d-d10631ff535e / ext4    defaults,usrquota 1 1
# reboot or remount
mount -o remount /
quotacheck -cmug /

Node: SELinux and System Control

Setup SELINUX Booleans 

setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_read_user_content=on httpd_enable_homedirs=on httpd_run_stickshift=on allow_polyinstantiation=on

Update selinux file setting 

restorecon -rv /var/run
restorecon -rv /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid
restorecon -rv /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift

SYSTEM CONTROL SETTINGS 

echo "# Added for OpenShift" >> /etc/sysctl.d/openshift.conf
echo "kernel.sem = 250  32000 32  4096" >> /etc/sysctl.d/openshift.conf
echo "net.ipv4.ip_local_port_range = 15000 35530" >> /etc/sysctl.d/openshift.conf
echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/openshift.conf
sysctl -p /etc/sysctl.d/openshift.conf

Node: SSH, Port Proxy, and Node application

SSH 

vi /etc/ssh/sshd_config
> AcceptEnv GIT_SSH

perl -p -i -e "s/^#MaxSessions .*$/MaxSessions 40/" /etc/ssh/sshd_config
perl -p -i -e "s/^#MaxStartups .*$/MaxStartups 40/" /etc/ssh/sshd_config

/bin/systemctl restart  sshd.service

PORT PROXY 

firewall-cmd --add-port=35531-65535/tcp
firewall-cmd --permanent --add-port=35531-65535/tcp
firewall-cmd --list-all

/bin/systemctl enable openshift-port-proxy.service
/bin/systemctl restart  openshift-port-proxy.service

NODE SETUP 

/bin/systemctl enable httpd.service
/bin/systemctl enable openshift-gears.service

vi /etc/openshift/node.conf
> PUBLIC_HOSTNAME="node.example.com"
> PUBLIC_IP="192.168.122.161" (Node IP Address)
> BROKER_HOST="192.168.122.220" (Broker IP Address)
> CLOUD_DOMAIN="example.com"

/etc/cron.minutely/openshift-facts

Node: Reboot

We need to reboot to load all the node stuff correctly 

reboot

Testing

Test on Broker (after node is back up)

Check Messaging 

mco ping

Should look like 

node.example.com                         time=239.51 ms

---- ping statistics ----
1 replies max: 239.51 min: 239.51 avg: 239.51

Check Broker 

curl -k -u demo:demo https://localhost/broker/rest/api

Should look like 

{"data":{"API":{"href":"https://localhost/broker/rest/api","method":"GET","optional_params":[],"rel":"API entry point","required_params":[]},"GET_ENVIRONMENT":{"href":"https://localhost/broker/rest/environment","method":"GET","optional_params":[],"rel":"Get environment information","required_params":[]},"GET_USER"
...
:id","type":"string","valid_options":[]}]}},"messages":[],"status":"ok","supported_api_versions":[1.0,1.1,1.2,1.3],"type":"links","version":"1.3"}

Check and Setup User 

yum -y install rubygem-rhc
LIBRA_SERVER=broker.example.com rhc setup

Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot) 

OpenShift Client Tools (RHC) Setup Wizard

This wizard will help you upload your SSH keys, set your application namespace, and
check that other programs like Git are properly installed.

The server's certificate is self-signed, which means that a secure connection can't be
established to 'broker.example.com'.

You may bypass this check, but any data you send to the server could be intercepted by
others.

Connect without checking the certificate? (yes|no): yes
Login to broker.example.com: demo
Password: ****
 
OpenShift can create and store a token on disk which allows to you to access the
server without using your password. The key is stored in your home directory and
should be kept secret.  You can delete the key at any time by running 'rhc logout'.
Generate a token now? (yes|no) no

Saving configuration to /root/.openshift/express.conf ... done

No SSH keys were found. We will generate a pair of keys for you.

    Created: /root/.ssh/id_rsa.pub

Your public SSH key must be uploaded to the OpenShift server to access code.  Upload
now? (yes|no)
yes

Since you do not have any keys associated with your OpenShift account, your new key
will be uploaded as the 'default' key.

Uploading key 'default' ... done

Checking for git ... found git version 1.8.2.1

Checking common problems .. done

Checking your namespace ... none

Your namespace is unique to your account and is the suffix of the public URLs we
assign to your applications. You may configure your namespace here or leave it blank
and use 'rhc create-domain' to create a namespace later.  You will not be able to
create applications without first creating a namespace.

Please enter a namespace (letters and numbers only) |<none>|: demoland

Create an app 

rhc domain show -p demo
rhc app create test1 diy-0.1 -p demo

Test on Local Machine (after node is back up)

Setup your machine to use broker as a name server (Note: This might mess up normal network operations.) 

vi /etc/resolve.conf
# At the first line put "nameserver *broker ip address*"
nameserver 192.168.122.220

Check and Setup User 

yum -y install rubygem-rhc
LIBRA_SERVER=broker.example.com rhc setup

Should look like (Note: Generate a token now? no - client can handle it, broker in F19 cannot) 

OpenShift Client Tools (RHC) Setup Wizard

This wizard will help you upload your SSH keys, set your application namespace, and
check that other programs like Git are properly installed.

The server's certificate is self-signed, which means that a secure connection can't be
established to 'broker.example.com'.

You may bypass this check, but any data you send to the server could be intercepted by
others.

Connect without checking the certificate? (yes|no): yes
Login to broker.example.com: demo
Password: ****
 
OpenShift can create and store a token on disk which allows to you to access the
server without using your password. The key is stored in your home directory and
should be kept secret.  You can delete the key at any time by running 'rhc logout'.
Generate a token now? (yes|no) no

Saving configuration to /root/.openshift/express.conf ... done

No SSH keys were found. We will generate a pair of keys for you.

    Created: /root/.ssh/id_rsa.pub

Your public SSH key must be uploaded to the OpenShift server to access code.  Upload
now? (yes|no)
yes

Since you do not have any keys associated with your OpenShift account, your new key
will be uploaded as the 'default' key.

Uploading key 'default' ... done

Checking for git ... found git version 1.8.2.1

Checking common problems .. done

Checking your namespace ... none

Your namespace is unique to your account and is the suffix of the public URLs we
assign to your applications. You may configure your namespace here or leave it blank
and use 'rhc create-domain' to create a namespace later.  You will not be able to
create applications without first creating a namespace.

Please enter a namespace (letters and numbers only) |<none>|: demoland

Create an app 

rhc domain show -p demo
rhc app create test2 diy-0.1 -p demo


Check App You should be able to go to the following URL in your web browser.

http://test2-demoland.example.com/

Retrieved from "https://fedoraproject.org/w/index.php?title=OpenShift_Origin-F19&oldid=348484"