User:Bcl

From FedoraProject

(Difference between revisions)
Jump to: navigation, search
(Building boot images)
(18 intermediate revisions by one user not shown)
Line 1: Line 1:
Newbie on the anaconda team
 
 
 
[http://www.brianlane.com Personal homepage]
 
[http://www.brianlane.com Personal homepage]
  
Maintainer of:
+
== Packages ==
 +
 
 
* [https://admin.fedoraproject.org/pkgdb/acls/name/pylint pylint] fedora package
 
* [https://admin.fedoraproject.org/pkgdb/acls/name/pylint pylint] fedora package
 
* [https://admin.fedoraproject.org/pkgdb/acls/name/python-logilab-common python-logilab-common] fedora package  
 
* [https://admin.fedoraproject.org/pkgdb/acls/name/python-logilab-common python-logilab-common] fedora package  
 
* [https://admin.fedoraproject.org/pkgdb/acls/name/python-logilab-astng python-logilab-astng] fedora package
 
* [https://admin.fedoraproject.org/pkgdb/acls/name/python-logilab-astng python-logilab-astng] fedora package
 +
* [https://admin.fedoraproject.org/pkgdb/acls/name/gnupg gnupg] fedora package (note, v1.4.10 not gnupg2)
 +
* [https://admin.fedoraproject.org/pkgdb/acls/name/livecd-tools livecd-tools] fedora package
 +
* [https://admin.fedoraproject.org/pkgdb/acls/name/parted parted] fedora package
 +
* [https://admin.fedoraproject.org/pkgdb/acls/name/bip bip] fedora package
 +
* [https://admin.fedoraproject.org/pkgdb/acls/name/mx mx] fedora package
 +
 +
== Misc ==
  
 +
* [http://bcl.fedorapeople.org/scripts/git-changelog git-changelog] A simplified version of dcantrell's [http://git.fedorahosted.org/git/?p=anaconda.git;a=blob;f=scripts/makebumpver;hb=HEAD makebumpver] script, it prints a nice summary of commits since the specified tag, suitable for use in a rpm spec file %changelog section.
 +
* [http://bcl.fedorapeople.org/ update images, logs, etc.]
  
 
== Anaconda Development ==
 
== Anaconda Development ==
  
 
Development System
 
Development System
* Fedora 12
+
* Fedora 18
* squid proxy to cache packages
+
* polipo proxy to cache packages
 
* lighttpd to serve up updates.img
 
* lighttpd to serve up updates.img
 
* tftpd to serve up pxe boot images
 
* tftpd to serve up pxe boot images
* mock + pungi for building
+
* mock or lorax for building packages and iso's
  
 
== mock setup ==
 
== mock setup ==
Line 37: Line 45:
 
** <code>mock -r fedora-13-i386 --init</code>
 
** <code>mock -r fedora-13-i386 --init</code>
 
** <code>mock -r fedora-13-i386 --no-clean --install pungi</code>
 
** <code>mock -r fedora-13-i386 --no-clean --install pungi</code>
* Enter the chroot - <code> mock -r fedora-13-i386 --shell </code>
+
 
** Edit <code> /usr/lib/anaconda-runtime/buildinstall </code>
+
Now you have a mock chroot environment setup that will use the proxy cache for packages.
*** add <code> proxy=http://proxy.home:3128 </code> to the main section of the yum config embedded in it.
+
 
** Edit <code> /usr/lib/python2.6/site-packages/pypungi/__init__.py </code> to add:
+
== Anaconda Debugging ==
 +
 
 +
=== pdb ===
 +
 
 +
Insert this into Anaconda at the point where you want to start debugging:
 +
* <code>import pdb; pdb.set_trace()</code>
 +
 
 +
When the breakpoint is hit you will be able to use pdb on tty1.
 +
 
 +
=== winpdb ===
 +
 
 +
http://winpdb.org is a remote python debugger with support for threads.
 +
 
 +
Insert this into Anaconda at the point where you want to start debugging:
 +
* <code>import rpdb2; rpdb2.start_embedded_debugger('snakes', fAllowRemote=True)</code>
 +
 
 +
Create an update image with the rpdb2.py file:
 +
* <code>./scripts/makeupdates -k</code>
 +
* <code>cp /usr/lib/python2.7/site-packages/rpdb2.py ./updates/tmp/updates/</code>
 +
* <code>./scripts/makeupdates -k</code>
 +
 
 +
Boot your iso with the updates=http://url/of/your/updates.img and change to tty2. Run <code>ip addr</code> to find the IP of the system/virt.
 +
 
 +
When Anaconda hits the breakpoint run winpdb on your local system and use File->Attach to connect to the system and start debugging. The password (snakes) is set by the start_embedded_debugger call.
 +
 
 +
Anaconda's storage imports appear to interfere with rpdb2's import wrapper so don't expect to be able to step through the whole program, but it is very useful for examining the state of the system and looking at the local variables for all the running threads.
 +
 
 +
 
 +
 
 +
== Building boot images ==
 +
 
 +
=== with lorax ===
 +
From the top level of a checked out lorax git repo I run:
 
<pre>
 
<pre>
diff --git a/src/pypungi/__init__.py b/src/pypungi/__init__.py
+
PATH=./src/sbin/:$PATH PYTHONPATH=./src/ ./src/sbin/lorax -p Fedora -v 17 -r 17 \
index e57e72a..d30eefa 100644
+
-s http://download.fedora.redhat.com/pub/fedora/linux/development/rawhide/x86_64/os/ \
--- a/src/pypungi/__init__.py
+
--proxy http://proxy.brianlane.com:3128 --config ./lorax.conf ./tmp/
+++ b/src/pypungi/__init__.py
+
@@ -206,6 +206,8 @@ class Pungi(pypungi.PungiBase):
+
                thisrepo.cost = repo.cost
+
            if repo.ignoregroups:
+
                thisrepo.enablegroups = 0
+
+            if repo.proxy:
+
+                thisrepo.proxy = repo.proxy
+
            self.ayum.repos.add(thisrepo)
+
            self.ayum.repos.enableRepo(thisrepo.id)
+
            self.ayum._getRepos(thisrepo=thisrepo.id, doSetup = True)
+
--  
+
1.6.6.1
+
 
</pre>
 
</pre>
  
This patch should be in the next release of pungi, but check to make sure just to be safe.
+
The -s points to repositories to use, add your own with higher nvr to override packages in the primary repo.
  
Now you have a mock chroot environment setup that will use the proxy cache for packages.
+
lorax.conf points to the template directory:
 +
<pre>
 +
[lorax]
 +
sharedir=/home/bcl/Red_Hat/projs/lorax/share
 +
</pre>
  
== Building boot images ==
+
results will be in ./tmp/
 +
 
 +
=== with pungi (old) ===
  
 
Use the compose and pungi.ks files below, place them into /root/ in the mock to be built. Replace REPO_URL_HERE with the same repo you selected for the mock's yum.conf file.
 
Use the compose and pungi.ks files below, place them into /root/ in the mock to be built. Replace REPO_URL_HERE with the same repo you selected for the mock's yum.conf file.
Line 104: Line 138:
 
* You need a git repo of Anaconda, I branch for the build so I don't clutter up master with build by-products.
 
* You need a git repo of Anaconda, I branch for the build so I don't clutter up master with build by-products.
 
** <code> git clone git://git.fedorahosted.org/anaconda.git </code>
 
** <code> git clone git://git.fedorahosted.org/anaconda.git </code>
** <code> cd anaconda; git checkout -b build-upates </code>
+
** <code> cd anaconda; git checkout -b build-updates </code>
* bind the anaconda directory to the mock being used (needs to be done as root)
+
* bind the anaconda directory to the mock being used by editing <code>/etc/mock/site-defaults.cfg</code> as described above.
** <code> mkdir /var/lib/mock/fedora-13-i386/root/root/anaconda/ </code>
+
** '''NOTE: DO NOT''' use <code>mount -o bind</code> to mount directories inside the mock. They will be obliterated the next time --init is run.
** <code> mount -o bind /home/bcl/projs/anaconda/ /var/lib/mock/fedora-13-i386/root/root/anaconda/ </code>
+
* Install the development tools group
 +
** <code> mock -r fedora-13-i386 --install @development-tools</code>
 +
** <code> mock -r fedora-13-i386 --install yum-utils</code>
 
* enter the chroot
 
* enter the chroot
 
** <code> mock -v -r fedora-13-i386 --shell </code>
 
** <code> mock -v -r fedora-13-i386 --shell </code>
** Install the development tools group
 
*** <code> yum groupinstall "development tools" </code>
 
 
** grab a copy of the current anaconda*src.rpm from someplace and install the dependencies with:
 
** grab a copy of the current anaconda*src.rpm from someplace and install the dependencies with:
 
*** <code> yum-builddep anaconda*src.rpm </code>
 
*** <code> yum-builddep anaconda*src.rpm </code>
Line 126: Line 160:
 
Now add <code> updates=http://proxy.home/updates/updates.img </code> to the kernel parameters when booting the install media.
 
Now add <code> updates=http://proxy.home/updates/updates.img </code> to the kernel parameters when booting the install media.
  
 +
NOTE: By running yum inside the chroot you may mess up the rpmdb version, this depends on how close your host system is to the target system. So YMMV
  
 
== Build with a test anaconda.rpm ==
 
== Build with a test anaconda.rpm ==
Line 133: Line 168:
 
Normally when a punji build is done it pulls anaconda from the repo/proxy cache. Instead you want it to use your new build (ie. when doing stage1 development which cannot be updated by updates= being passed to the kernel)
 
Normally when a punji build is done it pulls anaconda from the repo/proxy cache. Instead you want it to use your new build (ie. when doing stage1 development which cannot be updated by updates= being passed to the kernel)
  
* <code> mount -o bind /home/bcl/Red_Hat/projs/anaconda/ /var/lib/mock/fedora-13-i386/root/root/anaconda/ </code>
+
* make sure mock is mounting your anaconda directory as described above. Again, do not use <code>mount -o bind</code> to do it.
 
* <code> mock -v -r fedora-13-i386-proxy --shell </code>
 
* <code> mock -v -r fedora-13-i386-proxy --shell </code>
 
* <code> cd /root/anaconda </code>
 
* <code> cd /root/anaconda </code>
 
* Removed the cached files. Otherwise it pull pull from there and not update to the latest
 
* Removed the cached files. Otherwise it pull pull from there and not update to the latest
 
** <code> rm -rf /13 </code>
 
** <code> rm -rf /13 </code>
** <code> rm -rf /extra/released/pungi/cache/local </code>
+
** <code> rm -rf /pungi/cache/local </code>
 
* <code> ./autogen.sh </code>
 
* <code> ./autogen.sh </code>
 
* <code> ./configure </code>
 
* <code> ./configure </code>
Line 160: Line 195:
 
== Update boot.iso with new anaconda rpm ==
 
== Update boot.iso with new anaconda rpm ==
  
After you have a working boot.iso you can easily update it with the files from the new anaconda rpm you built above. Use the [[http://bcl.fedorapeople.org/scripts/upd_bootiso | upd_bootiso]] script to do this:
+
After you have a working boot.iso you can easily update it with the files from the new anaconda rpm you built above. Use the [[http://git.fedorahosted.org/git?p=anaconda.git;a=blob_plain;f=scripts/upd-bootiso;hb=HEAD upd_bootiso]] script to do this:
  
 
<code>upd_bootiso boot.iso anaconda-13.35-1.fc13.i686.rpm</code>
 
<code>upd_bootiso boot.iso anaconda-13.35-1.fc13.i686.rpm</code>
Line 169: Line 204:
  
 
Note, this script currently only works with x86 due to the fact that I borrowed part of it from the <code>mk-images.x86</code> script and haven't built on other architectures yet.
 
Note, this script currently only works with x86 due to the fact that I borrowed part of it from the <code>mk-images.x86</code> script and haven't built on other architectures yet.
 +
 +
== Releasing anaconda ==
 +
 +
This requires permission to upload new anaconda files and a [https://fedora.transifex.net/projects/p/anaconda/ Transifex account] with access to the Anaconda project. The transifex client needs to be setup on your system. This is described in the [http://git.fedorahosted.org/git/?p=anaconda.git;a=blob;f=docs/transifex.txt;hb=HEAD doc/transifex.txt] file in the anaconda source tree:
 +
 +
* git clean -d -x -f
 +
* <code>./autogen.sh && ./configure</code>
 +
** If that fails for deps: <code>yum-builddep anaconda.src.rpm</code>
 +
** Alternatively grab the srpm using <code> fedpkg srpm </code> and run yum-builddep on that.
 +
* <code> make bumpver </code>
 +
* Make sure anaconda.spec.in looks ok
 +
* <code> git commit -m "New version." </code>
 +
* <code> make release </code>
 +
* <code> git push </code>
 +
* <code> git push --tags </code>
 +
 +
In the fedora package then do:
 +
* <code> fedpkg switch-branch fXX </code>
 +
* Copy the anaconda.spec and anaconda-*tar.gz over to the package directory
 +
* Make sure anaconda.spec looks ok
 +
* <code> fedpkg new-sources anaconda-XX.XX-X.tar.gz </code>
 +
* <code> fedpkg clog </code>
 +
* Edit the clog to fix the first line, make it consistent with the "- [text]" changelog entries.
 +
* <code> git add anaconda.spec </code>
 +
* <code> git commit -F clog </code>
 +
* <code> git push </code>
 +
* <code> fedpkg build </code>
 +
* <code> fedpkg update </code>

Revision as of 15:39, 30 November 2012

Personal homepage

Contents

Packages

Misc

Anaconda Development

Development System

  • Fedora 18
  • polipo proxy to cache packages
  • lighttpd to serve up updates.img
  • tftpd to serve up pxe boot images
  • mock or lorax for building packages and iso's

mock setup

  • yum install mock
  • Edit the /etc/mock/site-defaults.cfg file to bind your working directories inside the mock root. This keeps mock from wiping those directories on --init
config_opts['plugin_conf']['bind_mount_enable'] = True
config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/home/user/projs/anaconda', '/root/anaconda' ))
config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/local/repo', '/repo' ))
config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/local/pungi', '/pungi' ))
  • Edit the /etc/mock/ file for the distribution or copy it to a new name and edit
    • Add proxy=http://proxy.home:3128 to the main section.
    • comment out mirrorlist entries
    • Open up the mirror list url(s) in a browser and pick a mirror, use the same one for base and updates
    • Add baseurl=<mirror url> in each section instead of mirrorlist
    • Do this for each section with a mirrorlist
  • Setup the initial mock environment, replace the fedora-13-i386 with the mock file edited above
    • mock -r fedora-13-i386 --init
    • mock -r fedora-13-i386 --no-clean --install pungi

Now you have a mock chroot environment setup that will use the proxy cache for packages.

Anaconda Debugging

pdb

Insert this into Anaconda at the point where you want to start debugging:

  • import pdb; pdb.set_trace()

When the breakpoint is hit you will be able to use pdb on tty1.

winpdb

http://winpdb.org is a remote python debugger with support for threads.

Insert this into Anaconda at the point where you want to start debugging:

  • import rpdb2; rpdb2.start_embedded_debugger('snakes', fAllowRemote=True)

Create an update image with the rpdb2.py file:

  • ./scripts/makeupdates -k
  • cp /usr/lib/python2.7/site-packages/rpdb2.py ./updates/tmp/updates/
  • ./scripts/makeupdates -k

Boot your iso with the updates=http://url/of/your/updates.img and change to tty2. Run ip addr to find the IP of the system/virt.

When Anaconda hits the breakpoint run winpdb on your local system and use File->Attach to connect to the system and start debugging. The password (snakes) is set by the start_embedded_debugger call.

Anaconda's storage imports appear to interfere with rpdb2's import wrapper so don't expect to be able to step through the whole program, but it is very useful for examining the state of the system and looking at the local variables for all the running threads.


Building boot images

with lorax

From the top level of a checked out lorax git repo I run:

PATH=./src/sbin/:$PATH PYTHONPATH=./src/ ./src/sbin/lorax -p Fedora -v 17 -r 17 \
-s http://download.fedora.redhat.com/pub/fedora/linux/development/rawhide/x86_64/os/ \
--proxy http://proxy.brianlane.com:3128 --config ./lorax.conf ./tmp/

The -s points to repositories to use, add your own with higher nvr to override packages in the primary repo.

lorax.conf points to the template directory:

[lorax]
sharedir=/home/bcl/Red_Hat/projs/lorax/share

results will be in ./tmp/

with pungi (old)

Use the compose and pungi.ks files below, place them into /root/ in the mock to be built. Replace REPO_URL_HERE with the same repo you selected for the mock's yum.conf file.

  • pungi.ks
# un-comment the applicable repo

repo --name=fedora --baseurl=REPO_URL_HERE --proxy=http://proxy.home:3128

# Very small install footprint
%packages
@base
kernel
syslinux
nomtools
anaconda
%end
  • compose
#!/bin/bash
echo "*** Running pungi --force --nosource --nodebuginfo --nosplitmedia -G -C -B -c /root/pungi.ks --ver=13"

pungi --cachedir=/pungi/cache --force --nosource --nodebuginfo --nosplitmedia -G -C -B -c /root/pungi.ks --ver=13
echo "*** Done."
  • chmod a+x compose
  • exit
  • mock -r fedora-13-i386 --chroot "/root/compose"

Change the --ver to match whichever release you are building for

Building updates

Anaconda includes the ability to update itself by passing updates=http://path/to/update.img to the kernel at boot time. This allows you to use the same boot media and test changes to stage2 of the installer.

  • This assumes a working mock chroot (ie. I built boot images with this one first)
  • You need a git repo of Anaconda, I branch for the build so I don't clutter up master with build by-products.
  • bind the anaconda directory to the mock being used by editing /etc/mock/site-defaults.cfg as described above.
    • NOTE: DO NOT use mount -o bind to mount directories inside the mock. They will be obliterated the next time --init is run.
  • Install the development tools group
    • mock -r fedora-13-i386 --install @development-tools
    • mock -r fedora-13-i386 --install yum-utils
  • enter the chroot
    • mock -v -r fedora-13-i386 --shell
    • grab a copy of the current anaconda*src.rpm from someplace and install the dependencies with:
      • yum-builddep anaconda*src.rpm
    • now build anaconda
      • ./autogen.sh
      • ./configure
      • make updates
    • exit
  • copy the updates.img over to the updates directory and optionally rename it
    • cp /var/lib/mock/fedora-13-i386/root/root/anaconda/updates.img /home/tftpboot/images/updates/

You can skip installing development tools and anaconda deps for subsequent iterations.

Now add updates=http://proxy.home/updates/updates.img to the kernel parameters when booting the install media.

NOTE: By running yum inside the chroot you may mess up the rpmdb version, this depends on how close your host system is to the target system. So YMMV

Build with a test anaconda.rpm

NOTE: This doesn't actually seem to work when the released anaconda version matches that in the branch. buildinstall isn't letting the local repo override the one in the repo

Normally when a punji build is done it pulls anaconda from the repo/proxy cache. Instead you want it to use your new build (ie. when doing stage1 development which cannot be updated by updates= being passed to the kernel)

  • make sure mock is mounting your anaconda directory as described above. Again, do not use mount -o bind to do it.
  • mock -v -r fedora-13-i386-proxy --shell
  • cd /root/anaconda
  • Removed the cached files. Otherwise it pull pull from there and not update to the latest
    • rm -rf /13
    • rm -rf /pungi/cache/local
  • ./autogen.sh
  • ./configure
  • make scratch
  • rpmbuild -ts --nodeps anaconda...tar.gz
  • yum-builddep /builddir/build/SRPMS/anaconda...src.rpm
  • rpmbuild --rebuild /builddir/build/SRPMS/anaconda...src.rpm
  • mkdir /repo
  • cp /builddir/build/RPMS/anaconda...rpm /repo
  • createrepo /repo
  • edit /root/pungi.ks and add
    • repo --name=local --baseurl=file:///repo/
  • exit
  • mock -v -r fedora-13-i386 --chroot "/root/compose"
  • rsync -avc /var/lib/mock/fedora-13-i386/root/13/i386/os/images/ /home/tftpboot/images/fedora/13/i386/

Boot with the pxe image and see what happens.

NOTE: The removal of the local repo cache is needed because yum gets confused by its presence.

Update boot.iso with new anaconda rpm

After you have a working boot.iso you can easily update it with the files from the new anaconda rpm you built above. Use the [upd_bootiso] script to do this:

upd_bootiso boot.iso anaconda-13.35-1.fc13.i686.rpm

This will extract the files from initrd.img and install.img on the boot.iso, update the files and then re-build the boot.iso

This is considerably faster than using mock + pungi to compose a whole new iso from scratch.

Note, this script currently only works with x86 due to the fact that I borrowed part of it from the mk-images.x86 script and haven't built on other architectures yet.

Releasing anaconda

This requires permission to upload new anaconda files and a Transifex account with access to the Anaconda project. The transifex client needs to be setup on your system. This is described in the doc/transifex.txt file in the anaconda source tree:

  • git clean -d -x -f
  • ./autogen.sh && ./configure
    • If that fails for deps: yum-builddep anaconda.src.rpm
    • Alternatively grab the srpm using fedpkg srpm and run yum-builddep on that.
  • make bumpver
  • Make sure anaconda.spec.in looks ok
  • git commit -m "New version."
  • make release
  • git push
  • git push --tags

In the fedora package then do:

  • fedpkg switch-branch fXX
  • Copy the anaconda.spec and anaconda-*tar.gz over to the package directory
  • Make sure anaconda.spec looks ok
  • fedpkg new-sources anaconda-XX.XX-X.tar.gz
  • fedpkg clog
  • Edit the clog to fix the first line, make it consistent with the "- [text]" changelog entries.
  • git add anaconda.spec
  • git commit -F clog
  • git push
  • fedpkg build
  • fedpkg update