From Fedora Project Wiki
No edit summary
No edit summary
 
(27 intermediate revisions by the same user not shown)
Line 15: Line 15:
not explain how to install the undercloud.  There are three machines in this scenario:
not explain how to install the undercloud.  There are three machines in this scenario:


{|
{| class="wikitable"
|arch
|-
|use
! arch
|portname1
! use
|MAC1
! portname1
|IP1
! MAC1
|portname2
! IP1
|MAC2
! portname2
|IP2
! MAC2
! IP2
|-
|-
|x86_64
|x86_64
|undercloud
||undercloud
|eno2
||eno2
|6c:ae:8b:29:2a:02
||6c:ae:8b:29:2a:02
|9.114.219.30
||9.114.219.30
|eno4
||eno4
|6c:ae:8b:29:2a:04
||6c:ae:8b:29:2a:04
|9.114.118.98
||9.114.118.98
|-
|-
|x86_64
|x86_64
|overcloud control
||overcloud control
|eth???
||enp12s0f0
|00:0a:f7:73:3c:c3
||5c:f3:fc:36:dd:68
|9.114.219.134
||9.114.219.134
|eth???
||eno3
|6c:ae:8b:29:27:fa
||6c:ae:8b:29:27:fb
|9.114.118.??? 13?
||9.114.118.??? 13?
|-
|-
|ppc64le
|ppc64le
|overcloud compute
||overcloud compute
|enP3p5s0f2
||enP3p5s0f2
|00:90:fa:74:05:52
||00:90:fa:74:05:52
|9.114.219.49
||9.114.219.49
|enP3p5s0f3
||enP3p5s0f3
|00:90:fa:74:05:53
||00:90:fa:74:05:53
|9.114.118.154
||9.114.118.154
|}
|}


Line 79: Line 80:
             "disk":"1000",
             "disk":"1000",
             "arch":"x86_64",
             "arch":"x86_64",
            "capabilities": "profile:control,boot_option:local",
             "pm_user":"user",
             "pm_user":"user",
             "pm_password":"update",
             "pm_password":"update",
Line 92: Line 94:
             "disk":"1000",
             "disk":"1000",
             "arch":"ppc64le",
             "arch":"ppc64le",
            "capabilities": "profile:compute,boot_option:local,arch_flavor:P8",
             "pm_password":"update",
             "pm_password":"update",
             "pm_addr":"9.114.118.155"
             "pm_addr":"9.114.118.155"
Line 103: Line 106:


<pre>
<pre>
[hamzy@pkvmci853 ~]$ (OCB=$(dig @192.168.122.1 -4 +short Overcloud.virbr0); UC=9.114.118.98; ssh-keygen -f ~/.ssh/known_hosts -R ${UC}; ssh-keyscan ${UC} >> ~/.ssh/known_hosts; scp -3 hamzy@${OCB}:~/ironic-python-agent.initramfs stack@${UC}:~/ppc64le-bm-deploy-ramdisk; scp -3 hamzy@${OCB}:~/ironic-python-agent.kernel stack@${UC}:~/ppc64le-bm-deploy-kernel; scp -3 hamzy@${OCB}:~/overcloud-full.qcow2 stack@${UC}:~/ppc64le-overcloud-full.qcow2; scp -3 hamzy@${OCB}:~/overcloud-full.initrd stack@${UC}:~/ppc64le-overcloud-full.initrd; scp -3 hamzy@${OCB}:~/overcloud-full.kernel stack@${UC}:~/ppc64le-overcloud-full.kernel)
(OCB=$(dig @192.168.122.1 -4 +short Overcloud.virbr0); UC=9.114.118.98; ssh-keygen -f ~/.ssh/known_hosts -R ${UC}; ssh-keyscan ${UC} >> ~/.ssh/known_hosts; scp -3 hamzy@${OCB}:~/ironic-python-agent.initramfs stack@${UC}:~/ppc64le-ironic-python-agent.initramfs; scp -3 hamzy@${OCB}:~/ironic-python-agent.kernel stack@${UC}:~/ppc64le-ironic-python-agent.kernel; scp -3 hamzy@${OCB}:~/overcloud-full.qcow2 stack@${UC}:~/ppc64le-overcloud-full.qcow2; scp -3 hamzy@${OCB}:~/overcloud-full.initrd stack@${UC}:~/ppc64le-overcloud-full.initrd; scp -3 hamzy@${OCB}:~/overcloud-full.kernel stack@${UC}:~/ppc64le-overcloud-full.kernel)
</pre>
</pre>


Line 109: Line 112:


<pre>
<pre>
[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/d558d6173970736fde63526da6824d391ada6042_89d45961/ironic-python-agent.tar | tar xvf -
[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/current-tripleo-rdo/ironic-python-agent.tar | tar xvf -
[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/d558d6173970736fde63526da6824d391ada6042_89d45961/overcloud-full.tar | tar xvf -
[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/current-tripleo-rdo/overcloud-full.tar | tar xvf -
[stack@oscloud5 ~]$ for FILE in ironic-python-agent.initramfs ironic-python-agent.kernel overcloud-full.initrd overcloud-full.vmlinuz overcloud-full.qcow2; do mv ${FILE} x86_64-${FILE}; done
</pre>
</pre>


Line 260: Line 262:
</pre>
</pre>


Prepare your system for some patches...
Ironic also needs patching to the ironic-inspector component.


<pre>
<pre>
[stack@oscloud5 ~]$ (cd /usr/share/openstack-tripleo-common/; sudo git init .; sudo find . | xargs sudo git add; sudo git commit -m "base")
[stack@oscloud5 ~]$ cat <<"EO_DNSMASQ_CFG" | sudo tee -a /etc/ironic-inspector/dnsmasq.conf
[stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleoclient/; sudo git init .; sudo find . -not -iname \*.pyc -and -not -iname \*.pyo | xargs sudo git add; sudo git commit -m "base")
# Client is ppc64le (OPAL) box and doesn't need a chain loader, but does need a custom config
[stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleo_common/; sudo git init .; sudo find . -not -iname \*.pyc -and -not -iname \*.pyo | xargs sudo git add; sudo git commit -m "base")
dhcp-match=set:ppc64le,option:client-arch,14
dhcp-boot=tag:ppc64le,
dhcp-option=tag:ppc64le, 209,/tftpboot/ppc64le/deploy.cfg
dhcp-mac=set:hamzy,6c:ae:8b:29:27:fa
dhcp-mac=set:hamzy,6c:ae:8b:29:27:fb
dhcp-mac=set:hamzy,00:90:fa:74:05:52
dhcp-mac=set:hamzy,00:90:fa:74:05:53
dhcp-mac=set:hamzy,00:0a:f7:73:3c:c2
dhcp-mac=set:hamzy,00:0a:f7:73:3c:c3
dhcp-ignore=tag:!hamzy
EO_DNSMASQ_CFG
[stack@oscloud5 ~]$ sudo mkdir /tftpboot/ppc64le; sudo chmod 0755 /tftpboot/ppc64le
[stack@oscloud5 ~]$ cat <<"EO_DEPLOY_CFG" | sudo tee /tftpboot/ppc64le/deploy.cfg
default IPA-Deploy
label IPA-Deploy
    kernel ::/tftpboot/ppc64le/agent.kernel
    initrd ::/tftpboot/ppc64le/agent.ramdisk
    append ipa-inspection-callback-url=http://9.114.118.98:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,numa-topology,logs systemd.journald.forward_to_console=yes ipa-debug=1 ipa-inspection-dhcp-all-interfaces=1 ipa-collect-lldp=1
    sysappend 0x02
EO_DEPLOY_CFG
[stack@oscloud5 ~]$ sudo cp ~stack/ppc64le-ironic-python-agent.kernel /tftpboot/ppc64le/agent.kernel
[stack@oscloud5 ~]$ sudo cp ~stack/ppc64le-ironic-python-agent.initramfs /tftpboot/ppc64le/agent.ramdisk
[stack@oscloud5 ~]$ sudo chmod 0644 /tftpboot/ppc64le/*; sudo chown -R ironic:ironic /tftpboot/ppc64le; sudo chcon -R system_u:object_r:tftpdir_t:s0 /tftpboot/ppc64le/
[stack@oscloud5 ~]$ sudo systemctl restart openstack-ironic-inspector-dnsmasq.service
</pre>
</pre>


There is a bug for needing the userid for machines using ipmi that needs to be patched around.
Find out what version of rpms that you are using.  Unfortunately, the patches do not apply cleanly for every release.  Look for the short git commit hash.


<pre>
<pre>
[stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleo_common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/0001-fix-usernameless-IPMI.patch | sudo git am)
[stack@oscloud5 ~]$ rpm -qf /usr/lib/python2.7/site-packages/tripleo_common/actions/baremetal.py
Applying: fix usernameless IPMI
openstack-tripleo-common-8.3.1-0.20180103233641.04317ff.el7.centos.noarch
[stack@oscloud5 ~]$ rpm -qf /usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_node.py
python-tripleoclient-8.1.1-0.20171231084754.52b71ea.el7.centos.noarch
</pre>
</pre>


There is a bug for hardcoded ironic-python-agent and overcloud names:
The short git commit hash for tripleo-common would be 04317ff and for python-tripleoclient would be 52b71ea.  Prepare your system for some patches...


<pre>
<pre>
[stack@oscloud5 ~]$ (cd /usr/share/openstack-tripleo-common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/0001-openstack-tripleo-common.patch | sudo git am)
[stack@oscloud5 ~]$ (VER=08; PYTHON_TRIPLEOCLIENT=52b71ea; TRIPLEO_COMMON=04317ff; git config --global user.name "Mark Hamzy"; git config --global user.email hamzy@us.ibm.com; git clone git://git.openstack.org/openstack/tripleo-common; git clone git://git.openstack.org/openstack/python-tripleoclient; cd ~/tripleo-common/; git checkout ${TRIPLEO_COMMON}; cd ~/python-tripleoclient/; git checkout ${PYTHON_TRIPLEOCLIENT})
Applying: openstack-tripleo-common
[stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleoclient/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/0001-tripleoclient.patch | sudo git am)
Applying: tripleoclient
[stack@oscloud5 ~]$ (cd /usr/lib/python2.7/site-packages/tripleo_common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/0001-tripleo_common.patch | sudo git am)
Applying: tripleo_common
</pre>
</pre>
There is a bug for needing the userid for machines using ipmi that needs to be patched around.  There are fixes to tripleo-common for hard coded ironic-python-agent and overcloud names.


<pre>
<pre>
[stack@undercloud ~]$ (for SERVICE in openstack-mistral-api.service openstack-mistral-engine.service openstack-mistral-executor.service; do sudo systemctl restart ${SERVICE}; done)
[stack@oscloud5 ~]$ (VER=08; TRIPLEO_COMMON=04317ff; cd tripleo-common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/${VER}.${TRIPLEO_COMMON}/0001-fix-usernameless-IPMI.patch | git am)
[stack@oscloud5 ~]$ for I in openstack-ironic-conductor.service openstack-ironic-inspector.service openstack-ironic-inspector-dnsmasq.service; do sudo systemctl restart ${I}; done
Applying: fix usernameless IPMI
[stack@oscloud5 ~]$ (VER=08; TRIPLEO_COMMON=04317ff; cd tripleo-common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/${VER}.${TRIPLEO_COMMON}/0001-Support-multiple-architectures.patch | git am)
Applying: Support multiple architectures
</pre>
 
There are fixes to python-tripleoclient for hard coded ironic-python-agent and overcloud names.
 
<pre>
[stack@oscloud5 ~]$ (VER=08; PYTHON_TRIPLEOCLIENT=52b71ea; cd python-tripleoclient/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/${VER}.${PYTHON_TRIPLEOCLIENT}/0001-Support-multiple-architectures.patch | git am)
Applying: Support multiple architectures
</pre>
</pre>


I then go through the process of uploading images for the overcloud:
After the patches have been applied, they need to be installed:


<pre>
<pre>
[stack@oscloud5 ~]$ source stackrc
[stack@oscloud5 ~]$ (cd python-tripleoclient/; sudo python setup.py install --force) 2>&1 | tee output.install.python-tripleoclient
(undercloud) [stack@oscloud5 ~]$ time openstack overcloud image upload
[stack@oscloud5 ~]$ (cd tripleo-common/; sudo python setup.py install --force) 2>&1 | tee output.install.tripleo-common
...
</pre>
</pre>


The overcloud-full qcow2 image needs to be recreated in glance so that it loses both the kernel_id and the ramdisk_id.  This way a full disk image can be deployed.  
Once everything has been patched, update the workbooks and restart all of the necessary services.


<pre>
<pre>
(undercloud) [stack@oscloud5 ~]$ (FILE="overcloud-full.qcow2"; UUID=$(openstack image list -f value | grep 'overcloud-full ' | awk '{print $1;}'); openstack image delete ${UUID}; openstack image create --container-format bare --disk-format qcow2 --min-disk 0 --min-ram 0 --file ${FILE} --public overcloud-full)
[stack@oscloud5 ~]$ (cd /usr/share/openstack-tripleo-common/; openstack workbook update workbooks/validations.yaml; openstack workbook update workbooks/baremetal.yaml)
[stack@oscloud5 ~]$ sudo mistral-db-manage --config-file /etc/mistral/mistral.conf --debug populate
[stack@oscloud5 ~]$ (for SERVICE in openstack-mistral-api.service openstack-mistral-engine.service openstack-mistral-executor.service; do sudo systemctl restart ${SERVICE}; done)
[stack@oscloud5 ~]$ for I in openstack-ironic-conductor.service openstack-ironic-inspector.service openstack-ironic-inspector-dnsmasq.service; do sudo systemctl restart ${I}; done
</pre>
</pre>


Now import the baremetal nodes and assign them profiles.
I then go through the process of uploading images for the overcloud:


<pre>
<pre>
(undercloud) [stack@oscloud5 ~]$ time openstack overcloud node import --provide instackenv.json 2>&1 | tee output.overcloud.node.import
[stack@oscloud5 ~]$ source stackrc
(undercloud) [stack@oscloud5 ~]$ openstack overcloud image upload
(undercloud) [stack@oscloud5 ~]$ AGENT_ARCH=ppc64le-P8 AGENT_NAME=${AGENT_ARCH}-ironic-python-agent time openstack overcloud image upload --whole-disk --os-image-name ppc64le-P8-overcloud-full.qcow2
(undercloud) [stack@oscloud5 ~]$ openstack image list
...
...
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
| 95fcda93-0be1-4868-aaeb-8b2713748757 | ppc64le-P8-bm-deploy-kernel  | active |
| Node UUID                            | Node Name | Provision State | Current Profile | Possible Profiles |
| e9ad6167-8d00-4c6f-a255-84e39682fdd5 | ppc64le-P8-bm-deploy-ramdisk  | active |
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
| 989b37c4-c529-466a-9dae-08830d06792c | ppc64le-P8-overcloud-full    | active |
| ff2fdac5-6cc5-47a9-a095-d942b3960795 |           | available      | None            |                  |
| 8d2fe1c2-604e-4097-b28c-662439c1aa58 | x86_64-bm-deploy-kernel      | active |
| ef3d7b3b-97b8-42ab-b501-896474df658f |           | available      | None            |                   |
| d8656461-cb0e-4743-9ec0-3c95f648984d | x86_64-bm-deploy-ramdisk      | active |
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
| 8307cc19-02e7-41e3-8077-42c552aa06d6 | x86_64-overcloud-full        | active |
(undercloud) [stack@oscloud5 ~]$ (COMPUTE=""; CONTROL=""; while IFS=$' ' read -r -a PROFILES; do NODE=${PROFILES[0]}; ARCH=$(openstack baremetal node show --fields properties --format shell ${NODE} | sed -r -e "s,^.*cpu_arch': u'([^']*).*$,\1,"); echo "ARCH=${ARCH} NODE=${NODE}"; if [ -z "${COMPUTE}" -a "${ARCH}" == "ppc64le" ]; then COMPUTE=${NODE}; echo "Compute:"; ironic node-update ${COMPUTE} replace properties/capabilities=profile:compute,boot_option:local; continue; elif [ -z "${CONTROL}" -a "${ARCH}" == "x86_64" ]; then CONTROL=${PROFILES[0]}; echo "Control:"; ironic node-update ${CONTROL} replace properties/capabilities=profile:control,boot_option:local; continue; else echo "Error: ${NODE}"; openstack baremetal node show --fields properties --format shell ${NODE}; fi; done < <(openstack overcloud profiles list -f value))
| bb233a3b-a3b5-486d-a79a-de010495d8d1 | x86_64-overcloud-full-initrd  | active |
| f3a0e21f-d6a8-48a9-be9b-51ae7d5b4855 | x86_64-overcloud-full-vmlinuz | active |
...
</pre>
 
<pre>
(undercloud) [stack@oscloud5 ~]$ openstack overcloud profiles list
(undercloud) [stack@oscloud5 ~]$ openstack overcloud profiles list
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
...
| Node UUID                            | Node Name | Provision State | Current Profile | Possible Profiles |
| 7f9470ab-b79c-4382-94a7-e4dc2ea7658c |          | available      | control         |                  |
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
| fdf15be5-2542-42ed-8252-a154c58a12bc |          | available      | compute         |                  |
| ff2fdac5-6cc5-47a9-a095-d942b3960795 |          | available      | compute         |                  |
...
| ef3d7b3b-97b8-42ab-b501-896474df658f |          | available      | control         |                  |
+--------------------------------------+-----------+-----------------+-----------------+-------------------+
</pre>
</pre>


Line 331: Line 372:
(undercloud) [stack@oscloud5 ~]$ cp -r /usr/share/openstack-tripleo-heat-templates templates
(undercloud) [stack@oscloud5 ~]$ cp -r /usr/share/openstack-tripleo-heat-templates templates
(undercloud) [stack@oscloud5 ~]$ (cd templates/; curl --silent -o - https://hamzy.fedorapeople.org/openstack-tripleo-heat-templates.patch | patch -p1)
(undercloud) [stack@oscloud5 ~]$ (cd templates/; curl --silent -o - https://hamzy.fedorapeople.org/openstack-tripleo-heat-templates.patch | patch -p1)
(undercloud) [stack@oscloud5 ~]$ time openstack overcloud deploy --debug --templates /home/stack/templates -e /home/stack/templates/environments/network-environment.yaml -e /home/stack/templates/environments/network-isolation-custom.yaml --control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute 2>&1 | tee output.overcloud.deploy
(undercloud) [stack@oscloud5 ~]$ cat << '__EOF__' > templates/environments/override-overcloud-image.yaml
parameter_defaults:
  NovaImage: ppc64le-overcloud-full
  ComputeImage: ppc64le-overcloud-full
  controllerImage: x86_64-overcloud-full
  ControllerImage: x86_64-overcloud-full
__EOF__
(undercloud) [stack@oscloud5 ~]$ time openstack overcloud deploy --templates /home/stack/templates -e /home/stack/templates/environments/network-environment.yaml -e /home/stack/templates/environments/network-isolation-custom.yaml -e /home/stack/templates/environments/override-overcloud-image.yaml --control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute 2>&1 | tee output.overcloud.deploy
</pre>
</pre>

Latest revision as of 19:53, 22 January 2018

On a CentOS7 system

[hamzy@oscloud5 ~]$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.3.1611 (Core) 
Release:        7.3.1611
Codename:       Core
[stack@oscloud5 ~]$ uname -a
Linux oscloud5.stglabs.ibm.com 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Unfortunately, it seems that Environment setup for baremetal environment does not explain how to install the undercloud. There are three machines in this scenario:

arch use portname1 MAC1 IP1 portname2 MAC2 IP2
x86_64 undercloud eno2 6c:ae:8b:29:2a:02 9.114.219.30 eno4 6c:ae:8b:29:2a:04 9.114.118.98
x86_64 overcloud control enp12s0f0 5c:f3:fc:36:dd:68 9.114.219.134 eno3 6c:ae:8b:29:27:fb 9.114.118.??? 13?
ppc64le overcloud compute enP3p5s0f2 00:90:fa:74:05:52 9.114.219.49 enP3p5s0f3 00:90:fa:74:05:53 9.114.118.154

So, following Undercloud installation, I perform the following:

[hamzy@oscloud5 ~]$ sudo useradd stack
[hamzy@oscloud5 ~]$ sudo passwd stack
[hamzy@oscloud5 ~]$ echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack
[hamzy@oscloud5 ~]$ sudo chmod 0440 /etc/sudoers.d/stack
[hamzy@oscloud5 ~]$ sudo su - stack
[stack@oscloud5 ~]$ sudo hostnamectl set-hostname oscloud5.stglabs.ibm.com
[stack@oscloud5 ~]$ sudo hostnamectl set-hostname --transient oscloud5.stglabs.ibm.com
[stack@oscloud5 ~]$ sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo
[stack@oscloud5 ~]$ sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo
[stack@oscloud5 ~]$ sudo yum install -y python-tripleoclient
[stack@oscloud5 ~]$ cp /usr/share/instack-undercloud/undercloud.conf.sample ~/undercloud.conf
[stack@oscloud5 ~]$ cat << '__EOF__' > instackenv.json
{
    "nodes": [
        {
            "pm_type":"pxe_ipmitool",
            "mac":[
                "5c:f3:fc:36:dd:68"
            ],
            "cpu":"2",
            "memory":"1048576",
            "disk":"1000",
            "arch":"x86_64",
            "capabilities": "profile:control,boot_option:local",
            "pm_user":"user",
            "pm_password":"update",
            "pm_addr":"9.114.219.33"
        },
        {
            "pm_type":"pxe_ipmitool",
            "mac":[
                "00:90:fa:74:05:53"
            ],
            "cpu":"16",
            "memory":"1048576",
            "disk":"1000",
            "arch":"ppc64le",
            "capabilities": "profile:compute,boot_option:local,arch_flavor:P8",
            "pm_password":"update",
            "pm_addr":"9.114.118.155"
        }
    ]
}
__EOF__

I transfer over the built ppc64le overcloud images:

(OCB=$(dig @192.168.122.1 -4 +short Overcloud.virbr0); UC=9.114.118.98; ssh-keygen -f ~/.ssh/known_hosts -R ${UC}; ssh-keyscan ${UC} >> ~/.ssh/known_hosts; scp -3 hamzy@${OCB}:~/ironic-python-agent.initramfs stack@${UC}:~/ppc64le-ironic-python-agent.initramfs; scp -3 hamzy@${OCB}:~/ironic-python-agent.kernel stack@${UC}:~/ppc64le-ironic-python-agent.kernel; scp -3 hamzy@${OCB}:~/overcloud-full.qcow2 stack@${UC}:~/ppc64le-overcloud-full.qcow2; scp -3 hamzy@${OCB}:~/overcloud-full.initrd stack@${UC}:~/ppc64le-overcloud-full.initrd; scp -3 hamzy@${OCB}:~/overcloud-full.kernel stack@${UC}:~/ppc64le-overcloud-full.kernel)

I then transfer over the built x86_64 overcloud images:

[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/current-tripleo-rdo/ironic-python-agent.tar | tar xvf -
[stack@oscloud5 ~]$ wget --quiet -O - https://images.rdoproject.org/pike/rdo_trunk/current-tripleo-rdo/overcloud-full.tar | tar xvf -

I then modify undercloud.conf as follows:

[stack@oscloud5 ~]$ cat << __EOF__ | patch -p0
--- undercloud.conf.orig        2017-08-25 12:04:54.935063830 +0000
+++ undercloud.conf 2017-08-25 12:05:17.561063576 +0000
@@ -17,21 +17,25 @@
 # defined by local_interface, with the netmask defined by the prefix
 # portion of the value. (string value)
 #local_ip = 192.168.24.1/24
+local_ip = 9.114.118.98/24
 
 # Network gateway for the Neutron-managed network for Overcloud
 # instances. This should match the local_ip above when using
 # masquerading. (string value)
 #network_gateway = 192.168.24.1
+network_gateway = 9.114.118.98
 
 # Virtual IP or DNS address to use for the public endpoints of
 # Undercloud services. Only used with SSL. (string value)
 # Deprecated group/name - [DEFAULT]/undercloud_public_vip
 #undercloud_public_host = 192.168.24.2
+undercloud_public_host = 9.114.118.98
 
 # Virtual IP or DNS address to use for the admin endpoints of
 # Undercloud services. Only used with SSL. (string value)
 # Deprecated group/name - [DEFAULT]/undercloud_admin_vip
 #undercloud_admin_host = 192.168.24.3
+undercloud_admin_host = 9.114.118.98
 
 # DNS nameserver(s) to use for the undercloud node. (list value)
 #undercloud_nameservers =
@@ -74,6 +78,7 @@
 # Network interface on the Undercloud that will be handling the PXE
 # boots and DHCP for Overcloud instances. (string value)
 #local_interface = eth1
+local_interface = eno4
 
 # MTU to use for the local_interface. (integer value)
 #local_mtu = 1500
@@ -82,18 +87,22 @@
 # instances. This should be the subnet used for PXE booting. (string
 # value)
 #network_cidr = 192.168.24.0/24
+network_cidr = 9.114.118.0/24
 
 # Network that will be masqueraded for external access, if required.
 # This should be the subnet used for PXE booting. (string value)
 #masquerade_network = 192.168.24.0/24
+masquerade_network = 9.114.118.0/24
 
 # Start of DHCP allocation range for PXE and DHCP of Overcloud
 # instances. (string value)
 #dhcp_start = 192.168.24.5
+dhcp_start = 9.114.118.240
 
 # End of DHCP allocation range for PXE and DHCP of Overcloud
 # instances. (string value)
 #dhcp_end = 192.168.24.24
+dhcp_end = 9.114.118.248
 
 # Path to hieradata override file. If set, the file will be copied
 # under /etc/puppet/hieradata and set as the first file in the hiera
@@ -112,12 +121,14 @@
 # doubt, use the default value. (string value)
 # Deprecated group/name - [DEFAULT]/discovery_interface
 #inspection_interface = br-ctlplane
+inspection_interface = br-ctlplane
 
 # Temporary IP range that will be given to nodes during the inspection
 # process.  Should not overlap with the range defined by dhcp_start
 # and dhcp_end, but should be in the same network. (string value)
 # Deprecated group/name - [DEFAULT]/discovery_iprange
 #inspection_iprange = 192.168.24.100,192.168.24.120
+inspection_iprange = 9.114.118.249,9.114.118.250
 
 # Whether to enable extra hardware collection during the inspection
 # process. Requires python-hardware or python-hardware-detect package
__EOF__

And install the undercloud:

[stack@oscloud5 ~]$ time openstack undercloud install 2>&1 | tee output.undercloud.install
...
Undercloud install complete.
...

Ironic needs some different settings to be able to support PXE for ppc64le:

[stack@oscloud5 ~]$ (cd /etc/ironic; cat << '__EOF__' | sudo patch -p0)
--- ironic.conf.orig    2017-11-03 13:54:06.440644679 +0000
+++ ironic.conf 2017-11-03 15:01:47.584431653 +0000
@@ -343,6 +343,7 @@
 # for this option to be unset. (string value)
 # Allowed values: debug, info, warning, error, critical
 #notification_level = <None>
+notification_level = debug
 
 # Directory where the ironic python module is installed.
 # (string value)
@@ -3506,7 +3507,7 @@
 # On ironic-conductor node, template file for PXE
 # configuration. (string value)
 #pxe_config_template = $pybasedir/drivers/modules/pxe_config.template
-pxe_config_template=$pybasedir/drivers/modules/ipxe_config.template
+pxe_config_template=$pybasedir/drivers/modules/pxe_config.template
 
 # On ironic-conductor node, template file for PXE
 # configuration for UEFI boot loader. (string value)
@@ -3517,6 +3518,7 @@
 # configuration per node architecture. For example:
 # aarch64:/opt/share/grubaa64_pxe_config.template (dict value)
 #pxe_config_template_by_arch =
+pxe_config_template_by_arch = ppc64le:$pybasedir/drivers/modules/pxe_config.template
 
 # IP address of ironic-conductor node's TFTP server. (string
 # value)
@@ -3547,7 +3549,7 @@
 
 # Bootfile DHCP parameter. (string value)
 #pxe_bootfile_name = pxelinux.0
-pxe_bootfile_name=undionly.kpxe
+pxe_bootfile_name=pxelinux.0
 
 # Bootfile DHCP parameter for UEFI boot mode. (string value)
 #uefi_pxe_bootfile_name = bootx64.efi
@@ -3556,10 +3558,11 @@
 # Bootfile DHCP parameter per node architecture. For example:
 # aarch64:grubaa64.efi (dict value)
 #pxe_bootfile_name_by_arch =
+pxe_bootfile_name_by_arch = ppc64le:config
 
 # Enable iPXE boot. (boolean value)
 #ipxe_enabled = false
-ipxe_enabled=True
+ipxe_enabled = false
 
 # On ironic-conductor node, the path to the main iPXE script
 # file. (string value)
__EOF__

Ironic also needs patching to the ironic-inspector component.

[stack@oscloud5 ~]$ cat <<"EO_DNSMASQ_CFG" | sudo tee -a /etc/ironic-inspector/dnsmasq.conf
# Client is ppc64le (OPAL) box and doesn't need a chain loader, but does need a custom config
dhcp-match=set:ppc64le,option:client-arch,14
dhcp-boot=tag:ppc64le,
dhcp-option=tag:ppc64le, 209,/tftpboot/ppc64le/deploy.cfg
dhcp-mac=set:hamzy,6c:ae:8b:29:27:fa
dhcp-mac=set:hamzy,6c:ae:8b:29:27:fb
dhcp-mac=set:hamzy,00:90:fa:74:05:52
dhcp-mac=set:hamzy,00:90:fa:74:05:53
dhcp-mac=set:hamzy,00:0a:f7:73:3c:c2
dhcp-mac=set:hamzy,00:0a:f7:73:3c:c3
dhcp-ignore=tag:!hamzy
EO_DNSMASQ_CFG
[stack@oscloud5 ~]$ sudo mkdir /tftpboot/ppc64le; sudo chmod 0755 /tftpboot/ppc64le
[stack@oscloud5 ~]$ cat <<"EO_DEPLOY_CFG" | sudo tee /tftpboot/ppc64le/deploy.cfg
default IPA-Deploy
label IPA-Deploy
    kernel ::/tftpboot/ppc64le/agent.kernel
    initrd ::/tftpboot/ppc64le/agent.ramdisk
    append ipa-inspection-callback-url=http://9.114.118.98:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,numa-topology,logs systemd.journald.forward_to_console=yes ipa-debug=1 ipa-inspection-dhcp-all-interfaces=1 ipa-collect-lldp=1
    sysappend 0x02
EO_DEPLOY_CFG
[stack@oscloud5 ~]$ sudo cp ~stack/ppc64le-ironic-python-agent.kernel /tftpboot/ppc64le/agent.kernel
[stack@oscloud5 ~]$ sudo cp ~stack/ppc64le-ironic-python-agent.initramfs /tftpboot/ppc64le/agent.ramdisk
[stack@oscloud5 ~]$ sudo chmod 0644 /tftpboot/ppc64le/*; sudo chown -R ironic:ironic /tftpboot/ppc64le; sudo chcon -R system_u:object_r:tftpdir_t:s0 /tftpboot/ppc64le/
[stack@oscloud5 ~]$ sudo systemctl restart openstack-ironic-inspector-dnsmasq.service

Find out what version of rpms that you are using. Unfortunately, the patches do not apply cleanly for every release. Look for the short git commit hash.

[stack@oscloud5 ~]$ rpm -qf /usr/lib/python2.7/site-packages/tripleo_common/actions/baremetal.py
openstack-tripleo-common-8.3.1-0.20180103233641.04317ff.el7.centos.noarch
[stack@oscloud5 ~]$ rpm -qf /usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_node.py
python-tripleoclient-8.1.1-0.20171231084754.52b71ea.el7.centos.noarch

The short git commit hash for tripleo-common would be 04317ff and for python-tripleoclient would be 52b71ea. Prepare your system for some patches...

[stack@oscloud5 ~]$ (VER=08; PYTHON_TRIPLEOCLIENT=52b71ea; TRIPLEO_COMMON=04317ff; git config --global user.name "Mark Hamzy"; git config --global user.email hamzy@us.ibm.com; git clone git://git.openstack.org/openstack/tripleo-common; git clone git://git.openstack.org/openstack/python-tripleoclient; cd ~/tripleo-common/; git checkout ${TRIPLEO_COMMON}; cd ~/python-tripleoclient/; git checkout ${PYTHON_TRIPLEOCLIENT})

There is a bug for needing the userid for machines using ipmi that needs to be patched around. There are fixes to tripleo-common for hard coded ironic-python-agent and overcloud names.

[stack@oscloud5 ~]$ (VER=08; TRIPLEO_COMMON=04317ff; cd tripleo-common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/${VER}.${TRIPLEO_COMMON}/0001-fix-usernameless-IPMI.patch | git am)
Applying: fix usernameless IPMI
[stack@oscloud5 ~]$ (VER=08; TRIPLEO_COMMON=04317ff; cd tripleo-common/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/${VER}.${TRIPLEO_COMMON}/0001-Support-multiple-architectures.patch | git am)
Applying: Support multiple architectures

There are fixes to python-tripleoclient for hard coded ironic-python-agent and overcloud names.

[stack@oscloud5 ~]$ (VER=08; PYTHON_TRIPLEOCLIENT=52b71ea; cd python-tripleoclient/; curl --silent -o - https://hamzy.fedorapeople.org/TripleO-multi-arch/${VER}.${PYTHON_TRIPLEOCLIENT}/0001-Support-multiple-architectures.patch | git am)
Applying: Support multiple architectures

After the patches have been applied, they need to be installed:

[stack@oscloud5 ~]$ (cd python-tripleoclient/; sudo python setup.py install --force) 2>&1 | tee output.install.python-tripleoclient
[stack@oscloud5 ~]$ (cd tripleo-common/; sudo python setup.py install --force) 2>&1 | tee output.install.tripleo-common

Once everything has been patched, update the workbooks and restart all of the necessary services.

[stack@oscloud5 ~]$ (cd /usr/share/openstack-tripleo-common/; openstack workbook update workbooks/validations.yaml; openstack workbook update workbooks/baremetal.yaml)
[stack@oscloud5 ~]$ sudo mistral-db-manage --config-file /etc/mistral/mistral.conf --debug populate
[stack@oscloud5 ~]$ (for SERVICE in openstack-mistral-api.service openstack-mistral-engine.service openstack-mistral-executor.service; do sudo systemctl restart ${SERVICE}; done)
[stack@oscloud5 ~]$ for I in openstack-ironic-conductor.service openstack-ironic-inspector.service openstack-ironic-inspector-dnsmasq.service; do sudo systemctl restart ${I}; done

I then go through the process of uploading images for the overcloud:

[stack@oscloud5 ~]$ source stackrc
(undercloud) [stack@oscloud5 ~]$ openstack overcloud image upload
(undercloud) [stack@oscloud5 ~]$ AGENT_ARCH=ppc64le-P8 AGENT_NAME=${AGENT_ARCH}-ironic-python-agent time openstack overcloud image upload --whole-disk --os-image-name ppc64le-P8-overcloud-full.qcow2
(undercloud) [stack@oscloud5 ~]$ openstack image list
...
| 95fcda93-0be1-4868-aaeb-8b2713748757 | ppc64le-P8-bm-deploy-kernel   | active |
| e9ad6167-8d00-4c6f-a255-84e39682fdd5 | ppc64le-P8-bm-deploy-ramdisk  | active |
| 989b37c4-c529-466a-9dae-08830d06792c | ppc64le-P8-overcloud-full     | active |
| 8d2fe1c2-604e-4097-b28c-662439c1aa58 | x86_64-bm-deploy-kernel       | active |
| d8656461-cb0e-4743-9ec0-3c95f648984d | x86_64-bm-deploy-ramdisk      | active |
| 8307cc19-02e7-41e3-8077-42c552aa06d6 | x86_64-overcloud-full         | active |
| bb233a3b-a3b5-486d-a79a-de010495d8d1 | x86_64-overcloud-full-initrd  | active |
| f3a0e21f-d6a8-48a9-be9b-51ae7d5b4855 | x86_64-overcloud-full-vmlinuz | active |
...
(undercloud) [stack@oscloud5 ~]$ openstack overcloud profiles list
...
| 7f9470ab-b79c-4382-94a7-e4dc2ea7658c |           | available       | control         |                   |
| fdf15be5-2542-42ed-8252-a154c58a12bc |           | available       | compute         |                   |
...

Patch the openstack-tripleo-heat-templates locally with our configuration and then do the deploy.

(undercloud) [stack@oscloud5 ~]$ cp -r /usr/share/openstack-tripleo-heat-templates templates
(undercloud) [stack@oscloud5 ~]$ (cd templates/; curl --silent -o - https://hamzy.fedorapeople.org/openstack-tripleo-heat-templates.patch | patch -p1)
(undercloud) [stack@oscloud5 ~]$ cat << '__EOF__' > templates/environments/override-overcloud-image.yaml
parameter_defaults:
  NovaImage: ppc64le-overcloud-full
  ComputeImage: ppc64le-overcloud-full
  controllerImage: x86_64-overcloud-full
  ControllerImage: x86_64-overcloud-full
__EOF__
(undercloud) [stack@oscloud5 ~]$ time openstack overcloud deploy --templates /home/stack/templates -e /home/stack/templates/environments/network-environment.yaml -e /home/stack/templates/environments/network-isolation-custom.yaml -e /home/stack/templates/environments/override-overcloud-image.yaml --control-scale 1 --compute-scale 1 --control-flavor control --compute-flavor compute 2>&1 | tee output.overcloud.deploy