Moji

Description

Moji is a simple build system in which the client grabs a package from the server, builds it locally using mock, and then uploads the results back to the original server. Moji stands for "mini koji" and it was written to support the Fedora-ARM pre-koji build effort (for example, F15-armv5tel).

Using Moji for the Fedora ARM Build Effort

Setting Up

To use a system as a Moji client for the ARM F15 armv5tel bringup:

1. Install the Moji repo file: rpm -Uvh http://scotland.proximity.on.ca/fedora-arm/moji-repo/rpm/moji-release-1-1.noarch.rpm
2. Install the Moji preconfigured client: yum install -y moji-client
3. Start the Moji server: service moji-client start
4. (Optional) Configure the Moji server to start at next boot: chkconfig moji-client on

Please see the arm@lists.fedoraproject.org mailing list for more information.

• Alternative link: moji-client

Starting and Stopping the Moji Client

• To start the Moji client: service moji-client start

• To keep the client active but in a state that does not accept new builds: service moji-client lock

• To "unlock" a "locked" client (note, restarting the service accomplishes the same thing) : service moji-client unlock

• To stop the Moji client immediately: service moji-client stop

Viewing Current Status

• Go to the moji server status page (http://moji.proximity.on.ca/moji)
• The repo dir containing all mock output for stage0 (http://australia.proximity.on.ca/fedora-arm/f15v5.0/repo/)

Moji Source Code

The source code for Moji is in the Fedora ARM git repository at http://fedorahosted.org/arm/ and can be viewed at https://fedorahosted.org/arm/browser/moji/

Application Information

Server

python /path/to/moji.py server /path/to/source /path/to/working /path/to/finished /path/to/repo [password]

• "source" is the folder containing source RPM files
  • the "source" folder can contain a file called "priority.txt" which is a simple plaintext file where each line is the filename of a source pkg that should be built first (ex. nled-2.52-7.fc15.src.rpm)
• "working" is a folder that contains any source RPM files currently being worked on by a builder
• "finished" is a folder containing source RPM files that have been built (failure or success)
• "repo" is the directory which contains the mock build result output of any "finished" builds
  • the "repo" directory should also be publicly accessible (ex. via httpd)
  • createrepo needs to also be run via some other means (ex. shell script via incrontab)

For the ARM F15 armv5tel startup, the Moji server is australia.proximity.on.ca.

Client

python /path/to/moji.py client <hostname or ip> <mock config name> [password]

• the mock option should be a config file name (ex fedora-15-arm)
  • the actual mock file (ex. /etc/mock/fedora-15-arm.cfg) should point to at least 2 repos: one to the moji server repo dir and another to a pre-established "main-line" repo

Design/Protocol

• defaults to TCP port 6789
• non-threaded and does not fork per child
  • does fork before any request that could take a while (ex. sending files or building packages)
• if a client disconnects, their build is considered failed and sent to another builder later on
• clients send dummy keep-alive ping messages thru its socket object
• more details to come soon...

Repo Management

$ vim repo.sh

• Note: Run the below script in a screen session

#!/bin/bash
export repo="/var/export/f15v5.0/repo"
export temp="/var/export/f15v5.0/temp"
while true
do
	chek=`ls -lt ${repo}/*/*.[rx][pm][ml] | head -n 1 | tail -n 1 | grep -i '.*\.rpm$'`
	if [ "${chek}" != "" ]
	then
		echo "Starting on [ `date` ]" | tee -a ~/repo.log
		
		mkdir "${temp}" 2> /dev/null # make sure the temp repo directory exists
		rm -fr ${temp}/* 2> /dev/null # clear the temp repo directory
		
		ls "${repo}" | while read line
		do
			ln -s "${repo}/${line}" "${temp}/${line}"
		done
		
		rm -fr ${temp}/repodata # remove any temp repo data links
		cp -fr ${repo}/repodata ${temp}/ # copy over the current repo data
		createrepo --update -d "${temp}" # create/update the new repo data
		cp -fr ${temp}/repodata ${repo}/ # copy back the new repo data
		
		echo "Ending on [ `date` ]" | tee -a ~/repo.log
	fi
	
	sleep 5 # sleep loop or risk fire!
done

Base Package Set

priority.txt

• Use these package names only to get a list of full package file names (ENVRA)
• Each package entry must be on its own line in the text file

acl attr audit audit-libs basesystem bash binutils bison byacc bzip2 bzip2-libs ca-certificates ccache check chkconfig chrpath cloog cloog-ppl cluster cmake compat-readline5 coreutils coreutils-libs corosync cpio cpp cracklib cracklib-dicts crda cronie cryptsetup-luks curl cvs cyrus-sasl cyrus-sasl-lib dash db4 db4-utils dbus dbus-glib dbus-python dev dhcp diffutils dos2unix doxygen e2fsprogs ed elfutils elfutils-libelf elfutils-libs expat expect fakechroot fakeroot fedora-release file file-libs filesystem findutils firebird flex fontconfig freetype gamin gawk gc gcc gcc-c+ gcc-host gcc-libgcc gd gdb gdbm gettext git glib2 glibc glibc-common glibc-devel glibc-headers gmp gperf gpm grep gstreamer guile gzip hostname icu imake info initscripts iproute iputils java-1.5.0-gcj kbd kernel kernel-headers keyutils keyutils-libs krb5 krb5-libs less libacl libattr libblkid libcap libcom_err libcurl libffi libgcc libgomp libidn libmount libmpc libselinux libsepol libssh2 libstdc++ libstdc++-devel libutempter libuuid libxml2 linux-atm llvm lm_sensors lua lvm2 lzo m2crypto m4 make man-db mingetty module-init-tools mpc mpfr ncurses ncurses-base ncurses-libs netpbm net-snmp net-tools nspr nss nss-softokn nss-softokn-freebl nss-sysinit nss-util oniguruma openldap openssh openssl pam passwd patch pcre perl php pinentry pkgconfig popt postgresql ppl procps psmisc psutils pth python python3 readline recode redhat-rpm-config rpm rpm-build rpm-libs rsync ruby sed setup shadow shadow-utils shared-mime-info sharutils sqlite startup-notification strace stunnel sysfsutils system-config-keyboard systemd system-setup-keyboard systemtap sysvinit tar t-binutils tcl tcp_wrappers tcsh teckit telnet texinfo texlive t-gcc tix tk tmpwatch tzdata u-boot udev unzip usbutils usermode ustr util-linux util-linux-ng uuid vim wget which xinetd x-loader xz xz-libs yum yum-metadata-parser zip zlib