Architectures/ARM/F15 Koji build

From FedoraProject

< Architectures | ARM(Difference between revisions)
Jump to: navigation, search
 
(11 intermediate revisions by one user not shown)
Line 1: Line 1:
 
This page aims to document and solidify the remaining steps in getting F15 built for both ARMv5 and ARMv7 in koji (and therefore available to users).
 
This page aims to document and solidify the remaining steps in getting F15 built for both ARMv5 and ARMv7 in koji (and therefore available to users).
  
== F15 strategy ==
+
== Fedora 15 ARM ==
  
ARMv7 has recently been the focus of build efforts with many thousands of packages now successfully built into stage3 and stage4. The plan is to move armv7 efforts into the same koji build farm as armv5. This introduces some challenges which are overcome in this strategy.
+
ARMv7 has recently been the focus of build efforts with many thousands of packages now successfully built into stage3 and stage4. The plan is to move ARMv7 efforts into the same koji build farm as ARMv5. This introduces some challenges which are overcome in this strategy.
  
=== ARMv5 external build ===
+
=== ARM v5/v7 Koji Build ===
  
ARMv5 must now build a package set that exactly matches that of ARMv7. The steps here are:
+
* Current Koji Build Status: http://arm.koji.fedoraproject.org/~jchiappetta/stats.html
# Working toolchain (coordinated by fossjon and Chris Tyler)
+
* Temp Koji External Repos: http://australia.proximity.on.ca/fedora-arm/15/ (arm & armhfp)
#* This is now done.
+
# "Stage 4" build from a build farm (external to koji). There is hardware available for this, and we envision it to be a relatively straightforward process, largely thanks to the armv7 work that has been taking place.
+
#* This is underway. Tracking site: http://australia.proximity.on.ca/moji/
+
#* Output repository: http://australia.proximity.on.ca/fedora-arm/f15v5.0/repo/
+
#* repoclosure report: http://australia.proximity.on.ca/paul/f15repoc.txt
+
#* comparison to armv7 repo: http://djdelorie.fedorapeople.org/armv5tel-vs-armv7hl.html
+
#* Pending tasks: finish all builds, remove differences between ARMv5 and ARMv7, finish repoclosure
+
  
When stage4 is ready and matching exactly for both architectures, the koji build can then start. Koji will be instructed to rebuild the exact stage4 package set, with the output builds going straight into koji.
+
=== Possible Problems ===
  
=== Stage4 cleanup ===
+
* Problems Noted: http://lists.fedoraproject.org/pipermail/arm/2011-December/002307.html
 +
* Meeting Notes + Problem Packages: https://lists.fedoraproject.org/pipermail/arm/2011-December/002326.html
 +
* Up-streaming any source RPMs that contain modifications (.armN packages)
 +
 
 +
=== Stage4 Cleanup ===
 +
 
 +
When stage4 is ready and matching exactly for both architectures, the koji build can then start. Koji will be instructed to rebuild the exact stage4 package set, with the output builds going straight into koji.
  
 
It is essential that stage4 is <em>pure</em>: that it only consists of packages from mainline Fedora 15 scm. GA packages are preferred, but packages from later points in the 'f15' git branches can be built when they are needed to fix ARM-specific bugs and build blockers.
 
It is essential that stage4 is <em>pure</em>: that it only consists of packages from mainline Fedora 15 scm. GA packages are preferred, but packages from later points in the 'f15' git branches can be built when they are needed to fix ARM-specific bugs and build blockers.
Line 29: Line 28:
 
DJ Delorie is tracking the divergence from Fedora here:  http://djdelorie.fedorapeople.org/armv7-srpms.html
 
DJ Delorie is tracking the divergence from Fedora here:  http://djdelorie.fedorapeople.org/armv7-srpms.html
  
=== Infrastructure preparation for armv7hl ===
+
=== ARMv7hl Infrastructure Preparation ===
  
 
As a new architecture, some core Fedora components must be updated to correctly support armv7hl.
 
As a new architecture, some core Fedora components must be updated to correctly support armv7hl.
Line 36: Line 35:
 
* <b>rpm</b>: A simple patch is needed, which could be applied only on armv7hl builds. Looks like Dennis is planning to take care of this.
 
* <b>rpm</b>: A simple patch is needed, which could be applied only on armv7hl builds. Looks like Dennis is planning to take care of this.
 
* <b>yum</b>: Yum must be extended to learn how to detect which ARM ABI is in use. This work is pending.
 
* <b>yum</b>: Yum must be extended to learn how to detect which ARM ABI is in use. This work is pending.
 +
 +
== Koji Que Script ==
 +
 +
* A list of stage4 source rpms were downloaded into a f15srpms/ folder
 +
 +
<pre>
 +
#!/bin/bash
 +
koji list-tagged dist-f15 > /tmp/k
 +
export l=`find ./f15srpms -type f | grep -i '\.src\.rpm' | wc -l`
 +
export x=1
 +
find ./f15srpms -type f | sort -R | grep -i '\.src\.rpm' > /tmp/p
 +
while true
 +
do
 +
ak list-tagged dist-f15 > /tmp/t
 +
while true
 +
do
 +
ak list-tasks --mine --quiet | grep '^[0-9]' | grep -Ei ' (open|free) .* build' > /tmp/n
 +
#echo "got tasks..." ; cat /tmp/n | wc -l ; echo
 +
if [ `cat /tmp/n | wc -l` -ge 10 ]
 +
then
 +
break
 +
fi
 +
p=`cat /tmp/p | head -n "$x" | tail -n 1`
 +
q=`basename "$p" | sed -e 's/[^0-9A-Za-z]/./g' -e 's/\.src\.rpm//g'`
 +
#echo "checking pkg [$p] [$q]..." ; echo
 +
c=`cat /tmp/n /tmp/t | grep -i "$q"`
 +
let x="($x % $l) + 1"
 +
if [ "$c" != "" ]
 +
then
 +
continue
 +
fi
 +
c=`cat /tmp/k | grep -i "$q"`
 +
if [ "$c" == "" ]
 +
then
 +
continue
 +
fi
 +
echo "queing [$p] skipped [$x]"
 +
ak build dist-f15 "$p" --nowait --background
 +
let n="$n + 1"
 +
done
 +
sleep 60
 +
done
 +
</pre>
  
 
== F16+ strategy ==
 
== F16+ strategy ==
  
 
The above approach of rebuilding the whole distro outside of koji on two architectures before getting koji to rebuild everything for the two architectures <em>again</em> is purely a one-off since we are adding an architecture (armv7) in the F15 cycle. The migration and building needed for F16 and onwards will be able to happen in much less complicated fashion, entirely inside koji.
 
The above approach of rebuilding the whole distro outside of koji on two architectures before getting koji to rebuild everything for the two architectures <em>again</em> is purely a one-off since we are adding an architecture (armv7) in the F15 cycle. The migration and building needed for F16 and onwards will be able to happen in much less complicated fashion, entirely inside koji.

Latest revision as of 17:56, 8 December 2011

This page aims to document and solidify the remaining steps in getting F15 built for both ARMv5 and ARMv7 in koji (and therefore available to users).

Contents

[edit] Fedora 15 ARM

ARMv7 has recently been the focus of build efforts with many thousands of packages now successfully built into stage3 and stage4. The plan is to move ARMv7 efforts into the same koji build farm as ARMv5. This introduces some challenges which are overcome in this strategy.

[edit] ARM v5/v7 Koji Build

[edit] Possible Problems

[edit] Stage4 Cleanup

When stage4 is ready and matching exactly for both architectures, the koji build can then start. Koji will be instructed to rebuild the exact stage4 package set, with the output builds going straight into koji.

It is essential that stage4 is pure: that it only consists of packages from mainline Fedora 15 scm. GA packages are preferred, but packages from later points in the 'f15' git branches can be built when they are needed to fix ARM-specific bugs and build blockers.

At the time of writing, stage4 includes a number of hacked packages (i.e. spec file modifications that were made to get ARM builds working that have not gone 'upstream' into f15). These must all be moved into stage3 and removed from stage4. Then they should be fixed properly, then rebuilt for stage4 from official scm sources. The fedora-arm community has a number of provenpackagers on hand who can commit the easy fixes, and the more complicated stuff should be resolved with assistance/oversight from package maintainers. In our F14 experience, package maintainers are generally very responsive to our efforts.

Such fixes must be made in the f15 branch of Fedora scm before fixed packages are built for stage4. It is not sufficient for them to be fixed only in F16 and/or rawhide. Similarly, when fixing f15 branches, please also make sure that f16 and rawhide branches are fixed. This will save us time when we move onto F16.

DJ Delorie is tracking the divergence from Fedora here: http://djdelorie.fedorapeople.org/armv7-srpms.html

[edit] ARMv7hl Infrastructure Preparation

As a new architecture, some core Fedora components must be updated to correctly support armv7hl.

  • koji: Dennis Gilmore is finishing and testing his patches
  • rpm: A simple patch is needed, which could be applied only on armv7hl builds. Looks like Dennis is planning to take care of this.
  • yum: Yum must be extended to learn how to detect which ARM ABI is in use. This work is pending.

[edit] Koji Que Script

  • A list of stage4 source rpms were downloaded into a f15srpms/ folder
#!/bin/bash
koji list-tagged dist-f15 > /tmp/k
export l=`find ./f15srpms -type f | grep -i '\.src\.rpm' | wc -l`
export x=1
find ./f15srpms -type f | sort -R | grep -i '\.src\.rpm' > /tmp/p
while true
do
	ak list-tagged dist-f15 > /tmp/t
	while true
	do
		ak list-tasks --mine --quiet | grep '^[0-9]' | grep -Ei ' (open|free) .* build' > /tmp/n
		#echo "got tasks..." ; cat /tmp/n | wc -l ; echo
		if [ `cat /tmp/n | wc -l` -ge 10 ]
		then
			break
		fi
		p=`cat /tmp/p | head -n "$x" | tail -n 1`
		q=`basename "$p" | sed -e 's/[^0-9A-Za-z]/./g' -e 's/\.src\.rpm//g'`
		#echo "checking pkg [$p] [$q]..." ; echo
		c=`cat /tmp/n /tmp/t | grep -i "$q"`
		let x="($x % $l) + 1"
		if [ "$c" != "" ]
		then
			continue
		fi
		c=`cat /tmp/k | grep -i "$q"`
		if [ "$c" == "" ]
		then
			continue
		fi
		echo "queing [$p] skipped [$x]"
		ak build dist-f15 "$p" --nowait --background
		let n="$n + 1"
	done
	sleep 60
done

[edit] F16+ strategy

The above approach of rebuilding the whole distro outside of koji on two architectures before getting koji to rebuild everything for the two architectures again is purely a one-off since we are adding an architecture (armv7) in the F15 cycle. The migration and building needed for F16 and onwards will be able to happen in much less complicated fashion, entirely inside koji.