From Fedora Project Wiki

No edit summary
No edit summary
 
(23 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Draft}}
== Description ==
== Description ==


Moji is a simple build system in which the client grabs a package from the server, builds it locally using mock, and then uploads the results back to the original server. Moji stands for "mini koji" and it was written to support the Fedora-ARM armv5tel F15 pre-koji build effort.
Moji is a simple build system in which the client grabs a package from the server, builds it locally using mock, and then uploads the results back to the original server. Moji stands for "mini koji" and it was written to support the Fedora-ARM pre-koji build effort (for example, F15-armv5tel).


== Using Moji for the Fedora ARM F15-armv5tel Effort ==
== Using Moji for the Fedora ARM Build Effort ==


=== Setting Up ===
=== Setting Up ===
To use a system as a Moji client for the ARM F15 armv5tel bringup:
To use a system as a Moji client for the ARM F15 armv5tel bringup:
# Install the Moji repo file: <code>rpm -Uvh http://scotland.proximity.on.ca/fedora-arm/moji-repo/moji-release.rpm</code>
# Install the Moji repo file: <code>rpm -Uvh http://scotland.proximity.on.ca/fedora-arm/moji-repo/rpm/moji-release-1-1.noarch.rpm</code>
# Install the Moji preconfigured client: <code>yum install -y moji</code>
# Install the Moji preconfigured client: <code>yum install -y moji-client</code>
# Start the Moji server: <code>service moji-client start</code>
# Start the Moji server: <code>service moji-client start</code>
# (Optional) Configure the Moji server to start at next boot: <code>chkconfig moji-client on</code>
# (Optional) Configure the Moji server to start at next boot: <code>chkconfig moji-client on</code>
Line 16: Line 14:
Please see the [https://admin.fedoraproject.org/mailman/listinfo/arm arm@lists.fedoraproject.org] mailing list for more information.
Please see the [https://admin.fedoraproject.org/mailman/listinfo/arm arm@lists.fedoraproject.org] mailing list for more information.


=== Stopping the Moji Client ===
* Alternative link: [http://australia.proximity.on.ca/jon/moji-client-0.1-10.fc13.noarch.rpm moji-client]
* To stop the Moji client gracefully, at the end of the current build: <code>service moji-client disable</code>
 
=== Starting and Stopping the Moji Client ===
* To start the Moji client: <code>service moji-client start</code>
 
* To keep the client active but in a state that does not accept new builds: <code>service moji-client lock</code>
 
* To "unlock" a "locked" client (note, restarting the service accomplishes the same thing) : <code>service moji-client unlock </code>
 
* To stop the Moji client immediately: <code>service moji-client stop</code>
* To stop the Moji client immediately: <code>service moji-client stop</code>


== Moji source code ==
=== Viewing Current Status ===
* Go to the moji server status page (http://moji.proximity.on.ca/moji)
* The repo dir containing all mock output for stage0 (http://australia.proximity.on.ca/fedora-arm/f15v5.0/repo/)
 
== Moji Source Code ==


The source code for Moji is in the Fedora ARM git repository at http://fedorahosted.org/arm/ and can be viewed at https://fedorahosted.org/arm/browser/moji/moji.py
The source code for Moji is in the Fedora ARM git repository at http://fedorahosted.org/arm/ and can be viewed at https://fedorahosted.org/arm/browser/moji/


== Application Information ==
== Application Information ==
Line 33: Line 42:


* "source" is the folder containing source RPM files
* "source" is the folder containing source RPM files
** the "source" folder can contain a file called "priority.txt" which is a simple plaintext file where each line is the filename of a source pkg that should be built first (ex. arpwatch-2.1a15-15.fc15.src.rpm)
** the "source" folder can contain a file called "priority.txt" which is a simple plaintext file where each line is the filename of a source pkg that should be built first (ex. nled-2.52-7.fc15.src.rpm)
* "working" is a folder that contains any source RPM files currently being worked on by a builder
* "working" is a folder that contains any source RPM files currently being worked on by a builder
* "finished" is a folder containing source RPM files that have been built (failure or success)
* "finished" is a folder containing source RPM files that have been built (failure or success)
Line 58: Line 67:
* if a client disconnects, their build is considered failed and sent to another builder later on
* if a client disconnects, their build is considered failed and sent to another builder later on
* clients send dummy keep-alive ping messages thru its socket object
* clients send dummy keep-alive ping messages thru its socket object
* more details to come soon...


==Repo Management==
==Repo Management==


===$ incrontab -e===
===$ vim repo.sh===
 
<pre>
/tmp/builds/done IN_CREATE touch /tmp/builds.flag
</pre>


===$ cat builds.sh===
* Note: Run the below script in a screen session


<pre>
<pre>
#!/bin/bash
#!/bin/bash
 
export repo="/var/export/f15v5.0/repo"
export temp="/var/export/f15v5.0/temp"
while true
while true
do
do
if [ -e /tmp/builds.flag ] # check the lock flag
chek=`ls -lt ${repo}/*/*.[rx][pm][ml] | head -n 1 | tail -n 1 | grep -i '.*\.rpm$'`
if [ "${chek}" != "" ]
then
then
echo "Starting on [ `date` ]" | tee -a /home/builder/builds.log
echo "Starting on [ `date` ]" | tee -a ~/repo.log
rm /tmp/builds.flag
mkdir "${temp}" 2> /dev/null # make sure the temp repo directory exists
mkdir /tmp/builds/repo 2> /dev/null # make sure the temp repo directory exists
rm -fr ${temp}/* 2> /dev/null # clear the temp repo directory
rm -frv /tmp/builds/repo/* 2> /dev/null # clear the repo directory
ls /tmp/builds/done | while read line
ls "${repo}" | while read line
do
do
if [ ! -e "/tmp/builds/repo/${line}" ]
ln -s "${repo}/${line}" "${temp}/${line}"
then
ln -s "/tmp/builds/done/${line}" "/tmp/builds/repo/${line}"
fi
done
done
rm -frv /tmp/builds/repo/repodata # remove any repo data links
rm -fr ${temp}/repodata # remove any temp repo data links
cp -frv /tmp/builds/done/repodata /tmp/builds/repo/ # copy over the old repo data
cp -fr ${repo}/repodata ${temp}/ # copy over the current repo data
createrepo --update -d /tmp/builds/repo # create/update the new repo data
createrepo --update -d "${temp}" # create/update the new repo data
cp -frv /tmp/builds/repo/repodata /tmp/builds/done/ # copy back the new repo data
cp -fr ${temp}/repodata ${repo}/ # copy back the new repo data
echo "Ending on [ `date` ]" | tee -a /home/builder/builds.log
echo "Ending on [ `date` ]" | tee -a ~/repo.log
fi
fi
Line 102: Line 106:
</pre>
</pre>


==Base Package Set (priority.txt)==
==Base Package Set==
 
===priority.txt===
 
* Use these package names only to get a list of full package file names (ENVRA)
* Each package entry must be on its own line in the text file


<pre>
<pre>
acl alsa-lib apr apr-util arts ascii aspell atk attr audiofile audit audit-libs avahi basesystem bash bc bind binutils bison byacc bzip2 bzip2-libs ca-certificates ccache cdparanoia celt check chkconfig chrpath cloog cloog-ppl cluster cmake compat-readline5 ConsoleKit coreutils coreutils-libs corosync cpio cpp cracklib cracklib-dicts crda cronie cryptsetup-luks cups curl cvs cyrus-sasl cyrus-sasl-lib dash db4 db4-utils dbus dbus-glib dbus-python deltarpm desktop-file-utils dev dhcp diffutils dos2unix doxygen dvipdfm dvipdfmx dvipng e2fsprogs ed elfutils elfutils-libelf elfutils-libs enchant esound expat expect fakechroot fakeroot fedora-release file file-libs filesystem findutils firebird flac flex fontconfig freeglut freetype gamin gawk gc gcc gcc-c+ gcc-host gcc-libgcc GConf2 gd gdb gdbm gdk-pixbuf2 gettext git glib2 glibc glibc-common glibc-devel glibc-headers gmp gnome-keyring gnupg gnupg2 gperf gpm grep gstreamer gtk2 gtk3 guile gzip hostname httpd hunspell icu imake info initscripts iproute iputils iw jasper java-1.5.0-gcj kbd kernel kernel-headers keyutils keyutils-libs krb5 krb5-libs lapack less lftp libacl libart_lgpl libassuan libasyncns libattr libavc1394 libblkid libbonobo libcap libcap-ng libcgroup libcom_er libconfig libcurl libdaemon libdbi libdmx libdrm libedit libffi libfontenc libfreebob libFS libgcc libgcrypt libglade2 libgomp libibverbs libICE libIDL libidn libiec61883 libieee1284 libjpeg-turbo libksba libmcrypt libmng libmodman libmount libmpc libnet libnl libnotify libogg liboil libotf libpaper libpciaccess libpng libraw1394 librdmacm librelp libselinux libsemanage libsepol libSM libsndfile libssh2 libstdc+ libstdc++-deve libtalloc libtasn1 libtdb libthai libtiff libtool libusb libusb1 libuser libutempter libuuid libvisual libvoikko libvorbis libX11 libXau libXaw libxcb libXcomposite libXcursor libXdamage libXdmcp libXevie libXext libxfce4util libXfixes libXfont libXft libXi libXinerama libxkbfile libxml2 libXmu libXNVCtrl libXp libXpm libXrandr libXrender libXres libXScrnSaver libxslt libXt libXtst libXv libXvMC libXxf86dga libXxf86misc libXxf86vm libyubikey linux-atm llvm lm_sensors lua lvm2 lynx lzo m2crypto m4 make man-db mingetty module-init-tools mpc mpfr mysql ncurses ncurses-base ncurses-libs netpbm net-snmp net-tools newt nkf nspr nss nss-softokn nss-softokn-freebl nss-sysinit nss-util oniguruma openjade openldap opensp openssh openssl ORBit2 pam pango passwd patch pcre pcsc-lite pcsc-lite-ccid perl perl-HTML-Parser perl-libs perl-Module-Pluggable perl-Pod-Escapes perl-Pod-Simple perl-threads perl-threads-shared perl-Tk perl-XML-Parser php pinentry pkgconfig plymouth polkit poppler popt postgresql ppl procps psmisc psutils pth pygpgme pyOpenSSL python python3 python-enchant python-krbV python-libs python-pycurl python-simplejson python-sqlalchemy PyXML qt3 rarian readline recode redhat-rpm-config rpm rpm-build rpm-libs rrdtool rsync ruby sed setup shadow shadow-utils shared-mime-info sharutils slang speex sqlite startup-notification strace stunnel sysfsutils system-config-keyboard systemd system-setup-keyboard systemtap sysvinit t1lib tar t-binutils tcl tcp_wrappers tcsh teckit telnet texinfo texlive t-gcc tidy tix tk tmpwatch transfig ttmkfdir tzdata u-boot udev unique3 unixODBC unzip usbutils usermode ustr util-linux util-linux-ng uuid vala vim wget which wireless-tools Xaw3d xcb-util xfconf xfig xinetd x-loader xmlto xorg-x11-drv-evdev xorg-x11-drv-fbdev xorg-x11-drv-keyboard xorg-x11-drv-v4l xorg-x11-drv-vesa xorg-x11-font-utils xorg-x11-utils xorg-x11-xauth xorg-x11-xdm xorg-x11-xfs xorg-x11-xfwp xorg-x11-xinit xorg-x11-xkb-utils xz xz-libs yum yum-metadata-parser zip zlib
acl attr audit audit-libs basesystem bash binutils bison byacc bzip2 bzip2-libs ca-certificates ccache check chkconfig chrpath cloog cloog-ppl cluster cmake compat-readline5 coreutils coreutils-libs corosync cpio cpp cracklib cracklib-dicts crda cronie cryptsetup-luks curl cvs cyrus-sasl cyrus-sasl-lib dash db4 db4-utils dbus dbus-glib dbus-python dev dhcp diffutils dos2unix doxygen e2fsprogs ed elfutils elfutils-libelf elfutils-libs expat expect fakechroot fakeroot fedora-release file file-libs filesystem findutils firebird flex fontconfig freetype gamin gawk gc gcc gcc-c+ gcc-host gcc-libgcc gd gdb gdbm gettext git glib2 glibc glibc-common glibc-devel glibc-headers gmp gperf gpm grep gstreamer guile gzip hostname icu imake info initscripts iproute iputils java-1.5.0-gcj kbd kernel kernel-headers keyutils keyutils-libs krb5 krb5-libs less libacl libattr libblkid libcap libcom_err libcurl libffi libgcc libgomp libidn libmount libmpc libselinux libsepol libssh2 libstdc++ libstdc++-devel libutempter libuuid libxml2 linux-atm llvm lm_sensors lua lvm2 lzo m2crypto m4 make man-db mingetty module-init-tools mpc mpfr ncurses ncurses-base ncurses-libs netpbm net-snmp net-tools nspr nss nss-softokn nss-softokn-freebl nss-sysinit nss-util oniguruma openldap openssh openssl pam passwd patch pcre perl php pinentry pkgconfig popt postgresql ppl procps psmisc psutils pth python python3 readline recode redhat-rpm-config rpm rpm-build rpm-libs rsync ruby sed setup shadow shadow-utils shared-mime-info sharutils sqlite startup-notification strace stunnel sysfsutils system-config-keyboard systemd system-setup-keyboard systemtap sysvinit tar t-binutils tcl tcp_wrappers tcsh teckit telnet texinfo texlive t-gcc tix tk tmpwatch tzdata u-boot udev unzip usbutils usermode ustr util-linux util-linux-ng uuid vim wget which xinetd x-loader xz xz-libs yum yum-metadata-parser zip zlib
</pre>
</pre>

Latest revision as of 20:51, 26 October 2011

Description

Moji is a simple build system in which the client grabs a package from the server, builds it locally using mock, and then uploads the results back to the original server. Moji stands for "mini koji" and it was written to support the Fedora-ARM pre-koji build effort (for example, F15-armv5tel).

Using Moji for the Fedora ARM Build Effort

Setting Up

To use a system as a Moji client for the ARM F15 armv5tel bringup:

  1. Install the Moji repo file: rpm -Uvh http://scotland.proximity.on.ca/fedora-arm/moji-repo/rpm/moji-release-1-1.noarch.rpm
  2. Install the Moji preconfigured client: yum install -y moji-client
  3. Start the Moji server: service moji-client start
  4. (Optional) Configure the Moji server to start at next boot: chkconfig moji-client on

Please see the arm@lists.fedoraproject.org mailing list for more information.

Starting and Stopping the Moji Client

  • To start the Moji client: service moji-client start
  • To keep the client active but in a state that does not accept new builds: service moji-client lock
  • To "unlock" a "locked" client (note, restarting the service accomplishes the same thing) : service moji-client unlock
  • To stop the Moji client immediately: service moji-client stop

Viewing Current Status

Moji Source Code

The source code for Moji is in the Fedora ARM git repository at http://fedorahosted.org/arm/ and can be viewed at https://fedorahosted.org/arm/browser/moji/

Application Information

Server

python /path/to/moji.py server /path/to/source /path/to/working /path/to/finished /path/to/repo [password]
  • "source" is the folder containing source RPM files
    • the "source" folder can contain a file called "priority.txt" which is a simple plaintext file where each line is the filename of a source pkg that should be built first (ex. nled-2.52-7.fc15.src.rpm)
  • "working" is a folder that contains any source RPM files currently being worked on by a builder
  • "finished" is a folder containing source RPM files that have been built (failure or success)
  • "repo" is the directory which contains the mock build result output of any "finished" builds
    • the "repo" directory should also be publicly accessible (ex. via httpd)
    • createrepo needs to also be run via some other means (ex. shell script via incrontab)

For the ARM F15 armv5tel startup, the Moji server is australia.proximity.on.ca.

Client

python /path/to/moji.py client <hostname or ip> <mock config name> [password]
  • the mock option should be a config file name (ex fedora-15-arm)
    • the actual mock file (ex. /etc/mock/fedora-15-arm.cfg) should point to at least 2 repos: one to the moji server repo dir and another to a pre-established "main-line" repo

Design/Protocol

  • defaults to TCP port 6789
  • non-threaded and does not fork per child
    • does fork before any request that could take a while (ex. sending files or building packages)
  • if a client disconnects, their build is considered failed and sent to another builder later on
  • clients send dummy keep-alive ping messages thru its socket object
  • more details to come soon...

Repo Management

$ vim repo.sh

  • Note: Run the below script in a screen session
#!/bin/bash
export repo="/var/export/f15v5.0/repo"
export temp="/var/export/f15v5.0/temp"
while true
do
	chek=`ls -lt ${repo}/*/*.[rx][pm][ml] | head -n 1 | tail -n 1 | grep -i '.*\.rpm$'`
	if [ "${chek}" != "" ]
	then
		echo "Starting on [ `date` ]" | tee -a ~/repo.log
		
		mkdir "${temp}" 2> /dev/null # make sure the temp repo directory exists
		rm -fr ${temp}/* 2> /dev/null # clear the temp repo directory
		
		ls "${repo}" | while read line
		do
			ln -s "${repo}/${line}" "${temp}/${line}"
		done
		
		rm -fr ${temp}/repodata # remove any temp repo data links
		cp -fr ${repo}/repodata ${temp}/ # copy over the current repo data
		createrepo --update -d "${temp}" # create/update the new repo data
		cp -fr ${temp}/repodata ${repo}/ # copy back the new repo data
		
		echo "Ending on [ `date` ]" | tee -a ~/repo.log
	fi
	
	sleep 5 # sleep loop or risk fire!
done

Base Package Set

priority.txt

  • Use these package names only to get a list of full package file names (ENVRA)
  • Each package entry must be on its own line in the text file
acl attr audit audit-libs basesystem bash binutils bison byacc bzip2 bzip2-libs ca-certificates ccache check chkconfig chrpath cloog cloog-ppl cluster cmake compat-readline5 coreutils coreutils-libs corosync cpio cpp cracklib cracklib-dicts crda cronie cryptsetup-luks curl cvs cyrus-sasl cyrus-sasl-lib dash db4 db4-utils dbus dbus-glib dbus-python dev dhcp diffutils dos2unix doxygen e2fsprogs ed elfutils elfutils-libelf elfutils-libs expat expect fakechroot fakeroot fedora-release file file-libs filesystem findutils firebird flex fontconfig freetype gamin gawk gc gcc gcc-c+ gcc-host gcc-libgcc gd gdb gdbm gettext git glib2 glibc glibc-common glibc-devel glibc-headers gmp gperf gpm grep gstreamer guile gzip hostname icu imake info initscripts iproute iputils java-1.5.0-gcj kbd kernel kernel-headers keyutils keyutils-libs krb5 krb5-libs less libacl libattr libblkid libcap libcom_err libcurl libffi libgcc libgomp libidn libmount libmpc libselinux libsepol libssh2 libstdc++ libstdc++-devel libutempter libuuid libxml2 linux-atm llvm lm_sensors lua lvm2 lzo m2crypto m4 make man-db mingetty module-init-tools mpc mpfr ncurses ncurses-base ncurses-libs netpbm net-snmp net-tools nspr nss nss-softokn nss-softokn-freebl nss-sysinit nss-util oniguruma openldap openssh openssl pam passwd patch pcre perl php pinentry pkgconfig popt postgresql ppl procps psmisc psutils pth python python3 readline recode redhat-rpm-config rpm rpm-build rpm-libs rsync ruby sed setup shadow shadow-utils shared-mime-info sharutils sqlite startup-notification strace stunnel sysfsutils system-config-keyboard systemd system-setup-keyboard systemtap sysvinit tar t-binutils tcl tcp_wrappers tcsh teckit telnet texinfo texlive t-gcc tix tk tmpwatch tzdata u-boot udev unzip usbutils usermode ustr util-linux util-linux-ng uuid vim wget which xinetd x-loader xz xz-libs yum yum-metadata-parser zip zlib