Architectures/ARM/Fedora15 HardFP Bootstrap
NOTE: This is a page in progress.
We are currently engaged in bootstrap of support for an ARMv7 (armv7hl) "hardfp" (hardware floating point) version of Fedora 15. In addition to adding support for hardware floating point, this port makes a transition to newer variant of the ARM ABI (sometimes referred to as a new variant of the "EABI" informally, though the name "EABI" does not exist any more in official terms for ARM systems), and in particular to a newer variant of the Procedure Calling Standard for ARM Architecture AAPCS. The newer ABI (as defined in section 6 of the AAPCS) uses floating point registers as part of the procedure calling interface. We will treat "armv7hl" (hardfp, little endian) as a new target architecture. As such, it is not supported to install both armv5tel (ARMv5) and armv7hl (ARMv7) packages on the same Fedora 15 system. We will, however, exploit the ability to run armv7hl within a chroot on an existing F13 ARM-based system in order to avoid more complicated bootstrap.
- Package status: Fedora15 HardFP Bootstrap package status
- Latest VFAD: July 15 2011 Bootstrapping Fedora Activity Day
- ssh://YOUR_FAS_USERNAME@git.fedorahosted.org/git/arm.git - armv7hl rootfs work in progress. This is suitable for use with "chroot" (after setting up a bind mount for /proc) from an existing ARM v7 compatible board running F13. If you are not in the gitarm FAS group within the Fedora account system, you can clone using anonymous git instead.
- git://fedorapeople.org/~djdelorie/bootstrap.git - upstream for stage1/stage2 scripts (we ultimately will sync the bits in the rootfs stage1/stage2 scripts here for future use).
We have various goals within this bootstrapping project. Briefly, these fall under "stages" (1,2,3,4,5) at which we anticipate certain levels of functionality from the root filesystem, beginning at a level of cross compilation and proceeding to native builds, and ultimately to a working mock and Koji configuration able to build production packages.
CURRENT STAGE: We are at stage3.
Stage 1 was an initial bootstrap using cross compilation (on x86_64 hosts) of a minimal set of armv7hl packages, from source (not from SRPMs), sufficient to have a minimal chroot environment from which gcc, binutils, and so forth were available for native builds. Stage 1 included gcc, binutils, glibc, etc. It did not include util-linux, or packages beyond the very minimal set required to get to a native building stage (still without RPM).
Stage 2 was an effort to get a working native build of RPM (rpmbuild) running within the rootfs. This included enough packages built from source (not from SRPMs, but on the target) to get a working "rpmbuild" binary sufficient to rebuild RPM and other packages. The output from this was an ability to build native RPM packages.
Stage 3 is an effort to get a minimal set of useful buildroot RPMs sufficient to rebuild the buildroot and have a working yum, mock configuration. Currently, we have build enough packages to run yum (python, etc.) - in fact many more, over 500 binary packages - but do not yet have all of the dependencies in place for mock. Although the binary runs, things like "userspace" need deps still, so we are investigating to ensure all of the deps are in place for that. We also need to continue building packages required for the minimal buildroot. The output from this stage will be hundreds of packages sufficient to rebuild all of the buildroot packages from within a working mock configuration.
At stage3, it is sufficient to intentionally disable certain unnecessary features of RPMs, and it is known that GNU autotools (and so on) will automatically exclude certain features dependent on non-present functionality. Consequently, these RPMs are not final and must be rebuilt in the following stages. While it is sufficient to disable sub-packages like "docs", or to turn off functionality like selinux, audit requirements, it is not sufficient to disable features that will be required by other packages directly in order to get a minimally working mock buildroot established. If in doubt, as on #fedora-arm for advice.
Output: RPMs and SRPMs in an armv7hl-YOUR_FAS_USERNAME bit branch of the armv7hl rootfs within the Fedora ARM git repo, stored in /stage3/armv7hl and later installed into the rootfs as part of merging by the maintainer (jonmasters). Mirrors of these packages are stored in a yum repo on scotland.proximity.on.ca/fedora-arm/armv7hl/
Stage 4 will be a point at which we are able to run a complete mock rebuild of all of the base buildroot packages. The output from this stage will be a working set of minimal RPMs that contain all of the base functionality for the distribution. These will be rebuilt again within a Koji environment in the next stage for reproducibility and verification.
There are various ways you can get involved. If in doubt, visit the Fedora ARM IRC (#fedora-arm on irc.freenode.net) and ask one of us for advice. You can also read more about our initial efforts to create a bootstrapped root filesystem in the git based workflow section here - please do help us by checking out the repo and following along on your board. We welcome your contribution(s), no matter how big or small to helping us get to a root filesystem sufficient for building with RPM.
- Jon Masters
- Chris Tyler
- Anthony Boccia
- Jon Chiappetta
- Paul Whalen
- Dennis Gilmore
- Brendan Conoboy
- Peter Robinson
Git based rootfs workflow
IMPORTANT: Please only perform these steps on real hardware. While emulators and cross-compilers are possible, we are doing a target-based bring up at this time. If you would like to make suggestions, please contact us. We have discussed the topic of emulation and cross-compilation many times before. We use cross-compilation to a certain extent in stage1, but we are trying to be very close to the "Fedora way" of operating, which is build on target. Therefore, please refrain from performing these steps from within a QEMU emulator, for example.
IMPORTANT: Build flags, and similar options are defined in the stage1 and stage2 scripts. In stage1/stage2, please do not enable features that are not required for minimal bootstrap, and especially DO NOT enable Thumb2, or NEON, and so forth in any contributed builds. In stage3 onward, do not change build flags, etc. as these are defined globally in the RPM packages (rpm, redhat-rpm-config, fedora-release...)
We have developed a workflow around a git based root filesystem. You will need to clone this git repository, either onto your board directory, or into a directory you export over NFS. In one possible situation, you may clone the git repository into a local "fedora_arm" repository, then clone this again into a root-owned "fedora_armv7hl_rootfs" that is NFS exported in such a fashion that it can be used as a chroot from an ARM v7 board running the existing Fedora 13 ARM release.
The following commands will clone the repository and setup the right branch(es):
$ git clone --branch armv7hl ssh://git.fedorahosted.org/git/arm.git fedora_armv7hl_rootfs $ cd fedora_armv7hl_rootfs $ git branch armv7hl-YOUR_FAS_USERNAME $ git checkout armv7hl-YOUR_FAS_USERNAME
- NOTE: Make sure that you have requested access to the Fedora ARM git repository in FAS, or just replace "ssh" with "git", and send patches/pull requests rather than committing directly to your own named branch of the repository - the use of "YOUR_FAS_USERNAME" is a convenience, it is not a technical requirement, we just do not want commits on master directly please, as it helps us track things more easily.
This will create a (your) named working branch of "armv7hl", which is a root filesystem. You should export this repository directly to your ARMv7 target board using NFS, or clone it again a second time remotely onto the target. A second clone can be beneficial because you will be running various commands inside this root filesystem under the root user account on the target and probably want to avoid permission problems with running commits as a regular user (otherwise, you will need to use "chown -R username" periodically to avoid problems managing the repo as a regular user). In the case of a second clone:
$ git clone --branch armv7hl-YOUR_FAS_USERNAME fedora_armv7hl_rootfs_target
- You can do stuff as root within this and "git push" back to your main repo without permission problems (ignore the lock warning during commit unless you are sharing this repository locally, in which case you need a sudo solution or similar)
IMPORTANT: Regardless of how you provide the target with the root filesystem, it does not contain a populated /dev directory due to needing to avoid having git need to house special files. This is done whenever you run stage2, but if you need to do it manually, you will need to perform these steps:
mkdir dev pushd dev sudo mknod null c 1 3 sudo mknod zero c 1 5 sudo mknod tty c 5 0 sudo mknod console c 5 1 sudo mknod sda b 8 0 sudo mknod sda1 b 8 1 sudo mknod sda2 b 8 2 sudo mknod sda3 b 8 3 sudo mknod sda4 b 8 4 sudo mknod mmcblk0 b 179 0 sudo mknod mmcblk0p1 b 179 1 sudo mknod mmcblk0p2 b 179 2 sudo mknod mmcblk0p3 b 179 3 sudo mknod mmcblk0p4 b 179 4 sudo mknod ttyO0 c 253 0 sudo mknod ttyO1 c 253 1 sudo mknod ttyO2 c 253 2 sudo mknod ttyO3 c 253 3 popd
Now that you have a working copy of the armv7hl root filesystem, and have exported it to your target either over NFS (probably from a second copy as in the above scenario) or by making a remote clone using git on/onto the target you should establish a correct /proc bind mount and "chroot" into this on your target (you may need to leave the chroot and re-enter if you use git commands that change the repository files remotely):
$ mount -o bind /proc /path/to/exported/fedora_armv7hl_rootfs_target/proc $ mount -o bind -t none /dev /path/to/exported/fedora_armv7hl_rootfs_target/dev $ mount -o bind -t none /dev/pts/ /path/to/exported/fedora_armv7hl_rootfs_target/dev/pts $ chroot /path/to/exported/fedora_armv7hl_rootfs_target/ /sbin/init
NOTE: The reason to bind mount /proc is that RPM contain logic to determine armv7hl compatibility that involves examining the cpuflags. If you do not bind mount /proc, you will build armv5tel binaries that are NOT ARMV7HL COMPATIBLE. Please make sure you can see /proc from within the chroot environment. A later set of patches to RPM may use alternative logic based around ELF file sectional attributes on RPM itself, but not yet.
Skip to "stage 3" below. The following is for historical interest.
Stage 2 bootstrap is now complete. But in this stage, the following was performed.
On the host, edit the "stage2/stage2" script, which is a copy of the one in the bootstrap repo. Please read and follow any instructions you find at the top of that script. In there, you will find various target "modules" (packages). Choose a package (refer to and update the real time list of work items at Etherpad) and add entries for it both to the latter part of the file, and a "go" command at the top to call it at the appropriate point during bootstrap (unless ordering matters, just as the last item in the current list). For example, there is a set of commands for db4 already in the current stage2 script:
NOTE: Make sure you updated the Etherpad with what you are working on
db4 ) mcd $BUILDDIR/db4 $SRC/db-*/dist/configure $TCONFIGARGS make $J make $J install ;;
There is a need to add a "go db4" higher in the file, in the correct position relative to the other "go" commands. In most cases, adding to the end of the list will be fine, unless ordering matters due to build dependencies - in that case, try to add a comment to this effect so that it is easier to understand why you have needed certain ordering.
On the host, you will need to download the most recent successfully built SRPM from primary arch Fedora 15 (refer to Koji to find the F15 SRPM for the package in question) to use as a seed. Extract and apply the patches required for Fedora through the use of the following commands (we don't currently keep a copy of the SRPM in the rootfs repo, since it is not really required - but we do keep the extracted sources, and so forth):
$ HOME=stage2/ rpm -ivh /path/to/downloaded/srpm $ HOME=stage2/ rpmbuild --target=armv7-unknown-linux-gnu --nodeps -bp stage2/rpmbuild/SPECS/package_name.spec
You should also add your package to the list of stage2 sources at the end of the stage1 script. If you have built your rootfs with stage1, you can then run "./stage1 stage2" to update the source tree in your stage2 area. If you have checked your stage2 area out from git, email your stage1 patch to DJ or Jon, and use the manual RPM instructions above to install your sources.
Ensure that, in any case (especially if you have overridden your RPM macros and the use of a HOME environment variable doesn't work for you) the extracted RPM sources, spec, and directory of fully applied bits are in place in stage2/rpmbuild (in SOURCES, SPECS, and BUILD, respectively). We will NOT change these from Fedora 15, nor make changes to them during build. All builds take place in a separate directory (stage2/builds), typically exploiting the fact that GNU Autoconf handles having sources in a separate directory from the build directory. If a particular build cannot operate in this fashion, use cp or rsync (see stage1 in the bootstrap repo for examples) to copy the bits into place in the builds/package_name directory.
After making changes to the stage2/stage2 script, you should be able to run the following from the target chroot:
$ /stage2/stage2 package_name
If you did things correctly, then the result should be a stage2/builds/package_name directory containing the build tree, and the bits should have been installed into the rootfs. If this works, please commit the changes onto your branch:
$ git commit -as new_paths_here
Then push these to any higher level repository you might have if you are working in a second copy (e.g. fedora_armv7hl_rootfs_target push to fedora_armv7hl_rootfs). Finally, with your changes on your armv7hl-YOUR_FAS_USERNAME branch, you can push this to an identically named branch on the Fedora ARM git repo, and request a merge by mailing the list (copy email@example.com with merge requests), or simply post a pull request from your externally hosted repo. You could try sending patches to the list directly, but they will likely be pretty large, so as a fallback, post a link to a patch file you are hosting elsewhere (such as fedorapeople) if you are unable to use git for this part.
Changes will be merged from various branches into the top-level armv7hl periodically. This ensures some ability to track who is making what changes, and when, and allows us to revert problematic builds. We will aim to keep this rootfs around only as long as we need it to get to the point where we have a working RPM.
The following things should be in your commit(s):
- Changes to stage2/stage2
- Additional exploded sources in stage2/rpmbuild
- The buildroot from building in stage/builds
- The installed bits in bin,etc,lib,usr,etc.
You should attach the exact version of the package you used (Fedora 15 NVR) in the commit message. If you require any clarification, refer to existing commits as examples for how to proceed and contact us on IRC.
Within the working chroot environment, you can see a /stage3 directory. This contains a directory named "armv7hl" filled with RPMs and SRPMs. You should choose a new package to build (using the armv7hl bootstrap package status wiki page for hints, and by asking on #fedora-arm if in doubt about useful packages to work on (or to verify nobody has started work and has not updated the wiki to indicate so doing). We need dependencies for mock (e.g. "userspace", and its dependencies) as well as other packages known to be in the minimal mock buildroots for Fedora or to be required to rebuild the minimal buildroot (a larger set).
Having chosen a packages, download its SRPM from Fedora 15 primary arch Koji. Stash that in /stage3/armv7hl/SRPMS. Then, from within /stage3:
$ HOME=`pwd` rpm -ivh armv7hl/SRPMS/the-package-srpm $ HOME=`pwd` rpmbuld -ba rpmbuild/SPECS/the-package.spec
For many packages, a "--nodeps" is required at the "rpmbuild" stage because its deps are available from stage1/stage2 binaries that are not yet rebuild in stage3. But verify that this is the case before forcing "--nodeps". Bonus points for rebuilding stage1/stage2 packages as real RPMs, and for noticing dependencies that are not explicitly provided. Do verify that dependencies have been built before building packages - if necessary, hold off on a package build and go build its dependencies recursively first.
The output from this exercise is an /stage3/armv7hl/RPMS and a /stage3/armv7hl/SRPMS update in a git repo branch named armv7hl-YOUR_FAS_USERNAME that can be pulled and merged into the rootfs. An alternative is to post a URL with your RPM/SRPM combination. In most cases the SRPM should be identical to the original F15 package. If you need to make changes, don't bump the release, but instead add "arm" to it (and "arm1", "arm2", etc. if needed), so creating foo-1.2-3arm rather than foo-1.2-3.
After requesting a merge, the armv7hl rootfs will be updated and your RPM(s) can be picked up by others for building dependent packages. The rootfs will explicitly contain an installed version which will be done as part of the merge (do not install your package directly before requesting a merge, since this may complicate merging).
We will be supporting the vfpv3-d16 architecture level as required of the optional AAPCS section 6 hardware floating point calling convention ABI. We will not be requiring vfpv3-d32 since not all ARMv7 parts implement all 32 double floating point registers and they are not required for base hardfp ABI compatibility. We will build for a Cortex-A8 optimized target, not requiring NEON, and with Thumb2 instructions disabled. Due to the way in which AAELF (ELF bindings for ARM Architecture) and AAPCS, as well as Thumb interworking are implemented, we can safely assume that Thumb2 can be enabled at the individual package level later without breaking any compatibility, or requiring other packages to be rebuilt.
Hardware floating point
Hardware floating point in ARMv7 processors takes the form of the VFPv3 (Vector Floating Point version 3) co-processor, which is a modern vector processing unit unrelated to the original ARM FPU present or emulated in earlier ARM systems.