User:Parasense/bodhi2-workflow

Bodhi2 push monitoring

• https://bodhi.fedoraproject.org/masher

Troubleshooting

# get the errors
sudo journalctl --since=yesterday -o short -u fedmsg-hub > ~/error.out
awk '/E[Rr][Rr]/' ~/error.out

# or with color
egrep 'ERROR|Errno' ~/error.out

# reset the fedmsg-hub service
sudo systemctl restart fedmsg-hub

OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-23-updates-x86_64/root/var/tmp/rpm-ostree.hjvMfC'

A bind mount needs to be removed. Look for TMPFS relics from rpm-ostree

findmnt -t tmpfs -o TARGET | grep rpm-ostree

Dismount ALL the relic TMPFS mounts (warning it's unwise to do this unless you know bodhi2 is completely idle)

sudo umount /var/lib/mock/*/root/var/tmp/rpm-ostree.??????

Signing

start the bodhi push for signing (responding 'no' when prompted)

cd /var/cache/sigul
yes 'no' | sudo -u masher -S bodhi-push --releases '23 22 21 5 6 7' --username parasense

# sign the push
for i in 23 22 21 ; do NSS_HASH_ALG_SUPPORT=+MD5 ~/releng/scripts/sigulsign_unsigned.py fedora-$i -v --write-all --sigul-batch-size=25 $(cat /var/cache/sigul/{Stable,Testing}-F${i})    ; done
for i in  7  6  5 ; do NSS_HASH_ALG_SUPPORT=+MD5 ~/releng/scripts/sigulsign_unsigned.py epel-$i   -v --write-all --sigul-batch-size=25 $(cat /var/cache/sigul/{Stable,Testing}-*EL-${i}) ; done

# Inspect the repo cache areas (optional)
# Areas to verify empty ?
/mnt/fedora_koji/koji/mash/updates/dist-5E-epel{,-testing}.repocache/repodata/
/mnt/fedora_koji/koji/mash/updates/dist-6E-epel{,-testing}.repocache/repodata/
/mnt/fedora_koji/koji/mash/updates/epel7{,-testing}.repocache/repodata
/mnt/fedora_koji/koji/mash/updates/f21-updates{,-testing}.repocache/repodata
/mnt/fedora_koji/koji/mash/updates/f22-updates{,-testing}.repocache/repodata
/mnt/fedora_koji/koji/mash/updates/f23-updates{,-testing}.repocache/repodata

Test if bodhi masher is running

These checks are not scientific, but may help inform.

Check for existing masher locks of currently running push, or failed previous push

ls -l /mnt/koji/mash/updates/MASHING-*

Check for running bodhi2 push (via masher)

pgrep -af /usr/bin/mash

Also check for rsync

pgrep -af rsync

Resuming failed push

# resume the push interactivly
sudo -u masher bodhi-push --resume --username parasense

# resume the push responding yes to everything
yes| sudo -u masher -S bodhi-push --resume --username parasense

# Follow the output of fedmsg-hub
sudo journalctl -o short -u fedmsg-hub -l -f

Sign Bridge Tasks

# monitor the signing on bridge for potential stalls from bodhi-backend
ssh -v -o'ControlPath=none' sign-bridge01 'tail -f /var/log/sigul_bridge.log'

# Verify the bridge is running or not
pgrep -af bridge.py

# Restart the bridge as necessary 
sudo pkill -f -9 bridge.py
sudo NSS_HASH_ALG_SUPPORT=+MD5 sigul_bridge -d -v -v

# Review the bridge.py output
tail -f /var/log/sigul_bridge.log

Stable push requests

Sometimes an urgent request to have something pushed to stable.

Here we had two lorax builds in the testing queue, and QA engineer requested they go to stable ASAP. So you have to header over to the bodhi2 web front end, revoke the "tasting push", then choose to push the build to stable. Once that is done the bodhi-push comment will permit the stable push.

sudo -u masher bodhi-push --releases '21 22 23' --request=stable  --builds 'lorax-21.34-1.fc21 lorax-22.13-1.fc22 freeipa-4.2.3-1.fc23' --username parasense

Testing push request

So there was a stable push in progress, and folks requesting testing push. The testing push uses a different lock file, so they will go in parallel with the stable push in progress.

sudo -u masher bodhi-push --releases '21 22 23' --request=testing  --username parasense