OpenShift Origin-F19

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
Line 22: Line 22:
 
  /bin/systemctl start  ntpd.service
 
  /bin/systemctl start  ntpd.service
  
=2== Configure Bind DNS =2==
+
= Setup and Configure Broker =
 +
== Configure Bind DNS ==
 
yum -y install bind bind-utils
 
yum -y install bind bind-utils
  
Line 157: Line 158:
  
  
=3== Configure the BROKER DHCP client and hostname =3==
+
== Configure the BROKER DHCP client and hostname ==
  
 
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
 
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
Line 165: Line 166:
 
echo "broker.example.com" > /etc/hostname
 
echo "broker.example.com" > /etc/hostname
  
=4== Installing and configuring MongoDB =4==
+
== Installing and configuring MongoDB ==
  
 
yum -y install mongodb-server
 
yum -y install mongodb-server
Line 183: Line 184:
 
> exit
 
> exit
  
=5== Installing and configuring QPID =5==
+
== Installing and configuring QPID ==
 
# Activemq on F19 isn't ready for production.  When it is, we'll use that
 
# Activemq on F19 isn't ready for production.  When it is, we'll use that
 
# For now let's use QPID with mcollective.
 
# For now let's use QPID with mcollective.
Line 196: Line 197:
 
/usr/bin/systemctl status qpidd.service
 
/usr/bin/systemctl status qpidd.service
  
=6== Installing and configuring MCollective client (QPID) =6==
+
== Installing and configuring MCollective client (QPID) ==
 
yum -y install mcollective-client
 
yum -y install mcollective-client
 
mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig
 
mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig
Line 221: Line 222:
 
EOF
 
EOF
  
=7== Installing and configuring the broker application =7==
+
== Installing and configuring the broker application ==
 
# When mcollective was updated to 2.2.3 it created a conflict with one of our components.
 
# When mcollective was updated to 2.2.3 it created a conflict with one of our components.
 
# We are working on fixing the conflict, but until then, do the following.
 
# We are working on fixing the conflict, but until then, do the following.
Line 260: Line 261:
 
VALID_GEAR_SIZES="small,medium"
 
VALID_GEAR_SIZES="small,medium"
  
=8== Configuring the broker plugins and MongoDB user accounts =8==
+
== Configuring the broker plugins and MongoDB user accounts ==
  
 
cp /usr/share/gems/gems/openshift-origin-auth-remote-user-*/conf/openshift-origin-auth-remote-user.conf.example /etc/openshift/plugins.d/openshift-origin-auth-remote-user.conf
 
cp /usr/share/gems/gems/openshift-origin-auth-remote-user-*/conf/openshift-origin-auth-remote-user.conf.example /etc/openshift/plugins.d/openshift-origin-auth-remote-user.conf
Line 310: Line 311:
  
  
=9== Install and configure the web console =9==
+
= Setup and Configure Node =
The web console is not in Fedora 19, skipping this step
+
== Initial setup/configure of the node host ==
 
+
=10== Initial setup/configure of the node host =10==
+
 
# ON NODE
 
# ON NODE
 
yum update
 
yum update
Line 342: Line 341:
 
nm-tool
 
nm-tool
  
=10a== Configure the NODE DHCP client and hostname =10a==
+
=== Configure the NODE DHCP client and hostname ===
 
# ON NODE
 
# ON NODE
 
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
 
echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf
Line 351: Line 350:
 
reboot
 
reboot
  
=11== Setting up MCollective on the node host =11==
+
== Setting up MCollective on the node host ==
 
# ON NODE
 
# ON NODE
 
yum -y install openshift-origin-msg-node-mcollective
 
yum -y install openshift-origin-msg-node-mcollective
Line 385: Line 384:
 
mco ping
 
mco ping
  
=12== Setting up node packages on the node host =12==
+
== Setting up node packages on the node host ==
 
# ON NODE
 
# ON NODE
 
yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util
 
yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util
Line 398: Line 397:
 
firewall-cmd --list-all
 
firewall-cmd --list-all
  
=13== Configuring PAM namespace module, cgropus, and user quotas on the node host =13==
+
== Configuring PAM namespace module, cgropus, and user quotas on the node host ==
 
# ON NODE
 
# ON NODE
 
# PAM
 
# PAM
Line 439: Line 438:
 
quotacheck -cmug /
 
quotacheck -cmug /
  
=14== Configuring SELinux and System Control on the node host =14==
+
== Configuring SELinux and System Control on the node host ==
 
# ON NODE
 
# ON NODE
 
# SELINUX
 
# SELINUX
Line 455: Line 454:
 
sysctl -p /etc/sysctl.d/openshift.conf
 
sysctl -p /etc/sysctl.d/openshift.conf
  
=14== Configuring SSH, Port Proxy, and Node on the node host =14==
+
== Configuring SSH, Port Proxy, and Node on the node host ==
 
# ON NODE
 
# ON NODE
 
# SSH
 
# SSH
Line 489: Line 488:
 
/etc/cron.minutely/openshift-facts
 
/etc/cron.minutely/openshift-facts
  
=14== Reboot Node and test =14==
+
== Reboot Node and test ==
 
# ON NODE
 
# ON NODE
 
reboot
 
reboot

Revision as of 17:23, 24 May 2013

Fedora 19 is when OpenShift Origin first became a feature.

This page is here to show how to setup OpenShift Origin on Fedora 19 using the packages in Fedora, as opposed to the packages published from upstream. These steps are written out to be done by hand. Yes, people can script and/or puppetize these steps. But these are written out so that people can see, and fine tune them.

Goal: By the end of this, you should have two machines. A broker machine, and one node machine. You should be able to create applications, that will be put on the node machine. You should be able to check the status of those applications. You should be able to point your web browser to the URL of those applications.

Note: There is no web console in Fedora 19. That will be in Fedora 20.

These instructions were created most from the following two places.

Contents

Initial Setup of Broker and Node Machines

ON BOTH BROKER AND NODE

# Start with a Fedora 19 minimal install
yum -y update
# avoid clock skew
yum -y install ntp
/bin/systemctl enable ntpd.service
/bin/systemctl start  ntpd.service

Setup and Configure Broker

Configure Bind DNS

yum -y install bind bind-utils

domain=example.com keyfile=/var/named/${domain}.key brokerip="$(nm-tool | grep Address | grep -v HW | awk '{print $2}')"

cd /var/named/ dnssec-keygen -a HMAC-MD5 -b 512 -n USER -r /dev/urandom ${domain} KEY="$(grep Key: K${domain}*.private | cut -d ' ' -f 2)" cd - rndc-confgen -a -r /dev/urandom echo $KEY

restorecon -v /etc/rndc.* /etc/named.* chown -v root:named /etc/rndc.key chmod -v 640 /etc/rndc.key

echo "forwarders { 8.8.8.8; 8.8.4.4; } ;" >> /var/named/forwarders.conf restorecon -v /var/named/forwarders.conf chmod -v 755 /var/named/forwarders.conf

rm -rvf /var/named/dynamic mkdir -vp /var/named/dynamic

echo $domain

cat <<EOF > /var/named/dynamic/${domain}.db \$ORIGIN . \$TTL 1 ; 1 seconds (for testing only) ${domain} IN SOA ns1.${domain}. hostmaster.${domain}. (

                        2011112904 ; serial
                        60         ; refresh (1 minute)
                        15         ; retry (15 seconds)
                        1800       ; expire (30 minutes)
                        10         ; minimum (10 seconds)
                         )
                    NS ns1.${domain}.
                    MX 10 mail.${domain}.

\$ORIGIN ${domain}. ns1 A 127.0.0.1

EOF

cat <<EOF > /var/named/${domain}.key key ${domain} {

 algorithm HMAC-MD5;
 secret "${KEY}";

}; EOF

cat /var/named/dynamic/${domain}.db cat /var/named/${domain}.key

chown -Rv named:named /var/named restorecon -rv /var/named

mv /etc/named.conf /etc/named.conf.openshift cat <<EOF > /etc/named.conf // named.conf // // Provided by Red Hat bind package to configure the ISC BIND named(8) DNS // server as a caching only nameserver (as a localhost DNS resolver only). // // See /usr/share/doc/bind*/sample/ for example named configuration files. //

options {

   listen-on port 53 { any; };
   directory "/var/named";
   dump-file "/var/named/data/cache_dump.db";
   statistics-file "/var/named/data/named_stats.txt";
   memstatistics-file "/var/named/data/named_mem_stats.txt";
   allow-query { any; };
   recursion yes;
   /* Path to ISC DLV key */
   bindkeys-file "/etc/named.iscdlv.key";
   // set forwarding to the next nearest server (from DHCP response
   forward only;
   include "forwarders.conf";

};

logging {

   channel default_debug {
       file "data/named.run";
       severity dynamic;
   };

};

// use the default rndc key include "/etc/rndc.key";

controls {

   inet 127.0.0.1 port 953
   allow { 127.0.0.1; } keys { "rndc-key"; };

};

include "/etc/named.rfc1912.zones";

include "${domain}.key";

zone "${domain}" IN {

   type master;
   file "dynamic/${domain}.db";
   allow-update { key ${domain} ; } ;

}; EOF

cat /etc/named.conf chown -v root:named /etc/named.conf restorecon /etc/named.conf

vi /etc/resolv.conf

  1. Add the following as the first nameserver in the file
  2. nameserver **your broker ip address**

firewall-cmd --add-service=dns firewall-cmd --permanent --add-service=dns firewall-cmd --list-all /bin/systemctl enable named.service /bin/systemctl start named.service

nsupdate -k ${keyfile} > server 127.0.0.1 > update delete broker.example.com A > update add broker.example.com 180 A **your broker ip address** > send > quit

ping broker.example.com dig @127.0.0.1 broker.example.com


Configure the BROKER DHCP client and hostname

echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf echo "supersede host-name \"broker\";" >> /etc/dhcp/dhclient-eth0.conf echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf

echo "broker.example.com" > /etc/hostname

Installing and configuring MongoDB

yum -y install mongodb-server

vi /etc/mongodb.conf

  1. Uncomment auth = true
  2. Add smallfiles = true

/usr/bin/systemctl enable mongod.service /usr/bin/systemctl status mongod.service /usr/bin/systemctl start mongod.service /usr/bin/systemctl status mongod.service

  1. Testing

mongo > show dbs > exit

Installing and configuring QPID

  1. Activemq on F19 isn't ready for production. When it is, we'll use that
  2. For now let's use QPID with mcollective.

yum install mcollective-qpid-plugin qpid-cpp-server firewall-cmd --add-port=5672/tcp firewall-cmd --permanent --add-port=5672/tcp firewall-cmd --list-all

/usr/bin/systemctl enable qpidd.service /usr/bin/systemctl start qpidd.service /usr/bin/systemctl status qpidd.service

Installing and configuring MCollective client (QPID)

yum -y install mcollective-client mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.orig

cat <<EOF > /etc/mcollective/client.cfg topicprefix = /topic/ main_collective = mcollective collectives = mcollective libdir = /usr/libexec/mcollective loglevel = debug logfile = /var/log/mcollective-client.log

  1. Plugins

securityprovider = psk plugin.psk = unset connector = qpid plugin.qpid.host=broker.example.com plugin.qpid.secure=false plugin.qpid.timeout=5

  1. Facts

factsource = yaml plugin.yaml = /etc/mcollective/facts.yaml EOF

Installing and configuring the broker application

  1. When mcollective was updated to 2.2.3 it created a conflict with one of our components.
  2. We are working on fixing the conflict, but until then, do the following.

yumdownloader openshift-origin-msg-common rpm -Uvh openshift-origin-msg-common-1.4.1-1.fc19.noarch.rpm --nodeps --force

yum -y install openshift-origin-broker openshift-origin-broker-util rubygem-openshift-origin-auth-remote-user rubygem-openshift-origin-msg-broker-mcollective rubygem-openshift-origin-dns-bind

sed -i -e "s/ServerName .*$/ServerName broker.example.com/" /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf cat /etc/httpd/conf.d/000002_openshift_origin_broker_servername.conf

/usr/bin/systemctl enable httpd.service /usr/bin/systemctl enable ntpd.service /usr/bin/systemctl enable sshd.service

firewall-cmd --add-service=ssh firewall-cmd --add-service=http firewall-cmd --add-service=https firewall-cmd --permanent --add-service=ssh firewall-cmd --permanent --add-service=http firewall-cmd --permanent --add-service=https firewall-cmd --list-all

openssl genrsa -out /etc/openshift/server_priv.pem 2048 openssl rsa -in /etc/openshift/server_priv.pem -pubout > /etc/openshift/server_pub.pem ssh-keygen -t rsa -b 2048 -f ~/.ssh/rsync_id_rsa cp -v ~/.ssh/rsync_id_rsa* /etc/openshift/

setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_run_stickshift=on named_write_master_zones=on fixfiles -R rubygem-passenger restore fixfiles -R mod_passenger restore restorecon -rv /var/run restorecon -rv /usr/share/gems/gems/passenger-*

vi /etc/openshift/broker.conf

  1. Might not have to do anything

CLOUD_DOMAIN="example.com" VALID_GEAR_SIZES="small,medium"

Configuring the broker plugins and MongoDB user accounts

cp /usr/share/gems/gems/openshift-origin-auth-remote-user-*/conf/openshift-origin-auth-remote-user.conf.example /etc/openshift/plugins.d/openshift-origin-auth-remote-user.conf cp /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf.example /etc/openshift/plugins.d/openshift-origin-msg-broker-mcollective.conf

domain=example.com keyfile=/var/named/${domain}.key cd /var/named/ KEY="$(grep Key: K${domain}*.private | cut -d ' ' -f 2)" cat $keyfile echo $KEY

cd /etc/openshift/plugins.d/ cat <<EOF > openshift-origin-dns-bind.conf BIND_SERVER="127.0.0.1" BIND_PORT=53 BIND_KEYNAME="${domain}" BIND_KEYVALUE="${KEY}" BIND_ZONE="${domain}" EOF

  1. pushd /usr/share/selinux/packages/rubygem-openshift-origin-dns-bind/ && make -f /usr/share/selinux/devel/Makefile ; popd
  2. semodule -i /usr/share/selinux/packages/rubygem-openshift-origin-dns-bind/dhcpnamedforward.pp

cp -v /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user-basic.conf.sample /var/www/openshift/broker/httpd/conf.d/openshift-origin-auth-remote-user.conf htpasswd -c -b -s /etc/openshift/htpasswd demo demopassword

  1. Don't forget your password. <demo password>

cat /etc/openshift/htpasswd

grep MONGO /etc/openshift/broker.conf mongo openshift_broker_dev --eval 'db.addUser("openshift", "mooo")'

  1. If you are going to change the username and/or password, change broker.conf

yum -y install rubygem-psych cd /var/www/openshift/broker

  1. This is being fixed, but for now do the following

vi Gemfile

  1. remove minitest version
  2. add gem 'psych'

gem install mongoid bundle --local

/usr/bin/systemctl enable openshift-broker.service /usr/bin/systemctl start httpd.service /usr/bin/systemctl start openshift-broker.service /usr/bin/systemctl status openshift-broker.service

curl -k -u demo:demopassword https://localhost/broker/rest/api


Setup and Configure Node

Initial setup/configure of the node host

  1. ON NODE

yum update yum -y install ntp /bin/systemctl enable ntpd.service /bin/systemctl start ntpd.service

  1. Find out the node ip address

nm-tool

  1. ON BROKER

domain=example.com keyfile=/var/named/${domain}.key

  1. Use the IP address from the node, found above

oo-register-dns -h node -d ${domain} -n 192.168.122.161 -k ${keyfile}

scp /etc/openshift/rsync_id_rsa.pub root@node.example.com:/root/.ssh/

  1. ON NODE

cat /root/.ssh/rsync_id_rsa.pub >> /root/.ssh/authorized_keys rm -f /root/.ssh/rsync_id_rsa.pub

  1. ON BROKER

ssh -i /root/.ssh/rsync_id_rsa root@node.example.com exit

  1. Find out the broker ip address

nm-tool

Configure the NODE DHCP client and hostname

  1. ON NODE

echo "prepend domain-name-servers **your broker ip address**;" >> /etc/dhcp/dhclient-eth0.conf echo "supersede host-name \"node\";" >> /etc/dhcp/dhclient-eth0.conf echo "supersede domain-name \"example.com\";" >> /etc/dhcp/dhclient-eth0.conf echo "node.example.com" > /etc/hostname

reboot

Setting up MCollective on the node host

  1. ON NODE

yum -y install openshift-origin-msg-node-mcollective mv /etc/mcollective/server.cfg /etc/mcollective/server.cfg.orig

cat <<EOF > /etc/mcollective/server.cfg topicprefix = /topic/ main_collective = mcollective collectives = mcollective libdir = /usr/libexec/mcollective logfile = /var/log/mcollective.log loglevel = debug daemonize = 1 direct_addressing = n

  1. Plugins

securityprovider = psk plugin.psk = unset connector = qpid plugin.qpid.host=broker.example.com plugin.qpid.secure=false plugin.qpid.timeout=5

  1. Facts

factsource = yaml plugin.yaml = /etc/mcollective/facts.yaml EOF

/bin/systemctl enable mcollective.service /bin/systemctl start mcollective.service

  1. ON BROKER

mco ping

Setting up node packages on the node host

  1. ON NODE

yum -y install rubygem-openshift-origin-node rubygem-passenger-native openshift-origin-port-proxy openshift-origin-node-util yum -y install openshift-origin-cartridge-cron-1.4 openshift-origin-cartridge-diy-0.1

firewall-cmd --add-service=ssh firewall-cmd --add-service=http firewall-cmd --add-service=https firewall-cmd --permanent --add-service=ssh firewall-cmd --permanent --add-service=http firewall-cmd --permanent --add-service=https firewall-cmd --list-all

Configuring PAM namespace module, cgropus, and user quotas on the node host

  1. ON NODE
  2. PAM

sed -i -e 's|pam_selinux|pam_openshift|g' /etc/pam.d/sshd

for f in "runuser" "runuser-l" "sshd" "su" "system-auth-ac" do

 t="/etc/pam.d/$f"
 if ! grep -q "pam_namespace.so" "$t"
 then
   echo -e "session\t\trequired\tpam_namespace.so no_unmount_on_close" >> "$t"
 fi

done

  1. CGROUPS
  2. echo "mount {" >> /etc/cgconfig.conf
  3. echo " cpu = /cgroup/all;" >> /etc/cgconfig.conf
  4. echo " cpuacct = /cgroup/all;" >> /etc/cgconfig.conf
  5. echo " memory = /cgroup/all;" >> /etc/cgconfig.conf
  6. echo " freezer = /cgroup/all;" >> /etc/cgconfig.conf
  7. echo " net_cls = /cgroup/all;" >> /etc/cgconfig.conf
  8. echo "}" >> /etc/cgconfig.conf
  9. restorecon -v /etc/cgconfig.conf
  10. mkdir /cgroup
  11. restorecon -RFvv /cgroup

/bin/systemctl enable cgconfig.service /bin/systemctl enable cgred.service /usr/sbin/chkconfig openshift-cgroups on /bin/systemctl restart cgconfig.service /bin/systemctl restart cgred.service /usr/sbin/service openshift-cgroups restart

  1. DISK QUOTA
  2. Edit fstab and add usrquota to whichever filesystem
  3. has /var/lib/openshift on it

UUID=b9e21eae-4b8c-4936-9f5d-d10631ff535e / ext4 defaults,usrquota 1 1

  1. reboot or remount

mount -o remount / quotacheck -cmug /

Configuring SELinux and System Control on the node host

  1. ON NODE
  2. SELINUX

setsebool -P httpd_unified=on httpd_can_network_connect=on httpd_can_network_relay=on httpd_read_user_content=on httpd_enable_homedirs=on httpd_run_stickshift=on allow_polyinstantiation=on

restorecon -rv /var/run restorecon -rv /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid restorecon -rv /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift

  1. SYSTEM CONTROL SETTINGS

echo "# Added for OpenShift" >> /etc/sysctl.d/openshift.conf echo "kernel.sem = 250 32000 32 4096" >> /etc/sysctl.d/openshift.conf echo "net.ipv4.ip_local_port_range = 15000 35530" >> /etc/sysctl.d/openshift.conf echo "net.netfilter.nf_conntrack_max = 1048576" >> /etc/sysctl.d/openshift.conf sysctl -p /etc/sysctl.d/openshift.conf

Configuring SSH, Port Proxy, and Node on the node host

  1. ON NODE
  2. SSH

vi /etc/ssh/sshd_config > AcceptEnv GIT_SSH

perl -p -i -e "s/^#MaxSessions .*$/MaxSessions 40/" /etc/ssh/sshd_config perl -p -i -e "s/^#MaxStartups .*$/MaxStartups 40/" /etc/ssh/sshd_config

/bin/systemctl restart sshd.service

  1. PORT PROXY

firewall-cmd --add-port=35531-65535/tcp firewall-cmd --permanent --add-port=35531-65535/tcp firewall-cmd --list-all

/bin/systemctl enable openshift-port-proxy.service /bin/systemctl restart openshift-port-proxy.service

  1. NODE SETUP

/bin/systemctl enable openshift-gears.service

  1. Find node and broker IP address

nm-tool

vi /etc/openshift/node.conf > PUBLIC_HOSTNAME="node.example.com" > PUBLIC_IP="192.168.122.161" (Node IP Address) > BROKER_HOST="192.168.122.220" (Broker IP Address) > CLOUD_DOMAIN="example.com"

/etc/cron.minutely/openshift-facts

Reboot Node and test

  1. ON NODE

reboot

  1. ON BROKER (after node is back up)

mco ping curl -k -u demo:demo https://localhost/broker/rest/api

yum -y install rubygem-rhc LIBRA_SERVER=broker.example.com rhc setup