FWN/Beats/Developments

Developments

In this section the people, personalities and debates on the @fedora-devel mailing list are summarized.

Contributing Writer: Oisin Feeley

Mysterious Fedora Compromise

The mysterious unavailability of much of the FedoraProject infrastructure (see FWN#139 "General Outage of Fedora Infrastructure"[1]) continued to provoke speculation during the week. Some light was shed[2] on 22-08-2008 when Paul Frields announced that there had been an intrusion onto several FedoraProject servers. The announcement emphasized that although one of the servers was used for signing rpm packages it was believed, based upon an absence of positive evidence, that the key and packages had not been tampered with. Nevertheless prudence and caution were being exercised and the opportunity was being seized to completely re-install and upgrade all systems. As a result it was necessary for most contributors to reset authentication tokens of various types (see this same issue[3].) It also appeared[4] that a concurrent event had led to the signing of some Red Hat OpenSSH packages, but that these had been quickly detected and had not led to the distribution of compromised packages.

[1] http://fedoraproject.org/wiki/FWN/Issue139#General_Outage_of_Fedora_Infrastructure

[2] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00012.html

[3] "Epiphany, Konqueror and Plague Hamper Updating SSL Client-side Certificates" [4] http://www.redhat.com/security/data/openssh-blacklist.html

Prior to that announcement all that was known was that there were problems on the build servers which became obvious to a wide audience on 13-08-2008 and that users were advised[5] on 14-08-2008 not to install updates. The wiki and email lists continued to function during this period thanks to the efforts of their administrators.

[5] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00008.html

An update[6] was posted on 18-08-2008 by Paul Frields that listed the services which had returned to normal and those which were expected to return to normal soon. Public speculation latched on[7][8] to the changing of "fedorahosted" SSH keys. Most guesses used this as evidence that something similar to the recent 2008 Debian OpenSSL vulnerabilities[9] had occurred. Some confusion prevailed[10] on @fedora-devel as to whether it was possible to trust the new key fingerprint on the website. Jim Meyering added[11] a useful post which explained how to change from using a DSA ssh key to an RSA ssh key. Overall there was a surprisingly low level of public discussion of the problem and it was not until 18-08-2008 that some complaints about the lack of information were expressed[12] on @fedora-list. On 22-08-2008 another bulletin was released[13] by Paul Frields that explained that the Fedora key had not been obviously compromised but that it was still being changed on the precautionary principle.

[6] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00011.html

[7] http://lwn.net/Articles/294547/

[8] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00790.html

[9] Metasploit has an excellent writeup on the topic here: http://www.metasploit.com/users/hdm/tools/debian-openssl/

[10] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00841.html

[11] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00845.html

[12] https://www.redhat.com/archives/fedora-list/2008-August/msg01953.html

[13] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00012.html

Although many responses to complaints about the limited information suggested[14] that the Fedora developers could be trusted to "do the right thing" in terms of alerting users to immediate threats of compromise there was still strong disquiet expressed[15] over the lack of information. This also occurred[16] on @fedora-list. Jef Spaleta wondered[17] if there might be a better way of getting information out than relying on everyone to subscribe to @fedora-announce. Alan Cox suggested[18] that an RSS feed for critical announcements could be picked up by a system's default package updater and that repositories could communicate errors to yum. Alan was also unhappy with the absence of important notices on the very front of the website and as a separate matter criticized the content of the information issued: "[...] leaving people in the dark assuming the worst [is] a very bad way to create long term trust." Bruno Wolf III also suggested[19] that information should have been conveyed by a more central announcement on fedoraproject.org and co-ordination with media such as LWN.net.

[14] https://www.redhat.com/archives/fedora-list/2008-August/msg01955.html

[15] https://www.redhat.com/archives/fedora-list/2008-August/msg02034.html

[16] https://www.redhat.com/archives/fedora-list/2008-August/msg02426.html

[17] https://www.redhat.com/archives/fedora-list/2008-August/msg02010.html

[18] https://www.redhat.com/archives/fedora-list/2008-August/msg02012.html

[19] https://www.redhat.com/archives/fedora-list/2008-August/msg02013.html

Most comments on @fedora-devel made a point of thanking the Fedora Infrastructure admins, even to the extent of providing internet beers[20] and Hans de Goede commented[21] that "Even before the unfortunate events of the last few weeks the infrastructure team has been doing an amazing job[.]"

[20] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01037.html

[21] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01028.html

Epiphany, Konqueror and Plague Hamper Updating SSL Client-side Certificates

As part of the general overhaul of Fedora Project infrastructure security occasioned by the recent intrusion[1] Dennis Gilmore advised[2] that the certificate authority governing access to cvs.fedoraproject.org and koji.fedoraproject.org had been changed. As a result it was necessary for those who wished to build packages or upload to the lookaside cache to manually import a new client-side certificate and to remove their old certificate.

[1] http://fedoraproject.org/wiki/FWN/Issue139#General_Outage_of_Fedora_Infrastructure

[2] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00962.html

Martin Sourada reported[3] that after following the procedure he still received a (Error code: ssl.error.unknown.ca.alert). Kai Engert suggested[4] that Martin might need to import the Fedora Project root CA certificate after verifying its fingerprint. As Martin had allowed exceptions for the Fedora websites this was not the problem and it turned out[5] to be due to a problem with the epiphany web browser failing to offer an option to remove old certificates.

[3] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00978.html

[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00982.html

[5] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00989.html

Problems were also experienced[6] by Jose Matos, but this time with the konqueror web browser (version 4.1.0). Kevin Koffler replied[7] that in KDE 4 there was no support for SSL certificate authentication with konqueror and pasted a link[8] to an upstream bug report filed with the KDE project on this issue.

[6] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00983.html

[7] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00999.html

[8] https://bugs.kde.org/show.bug.cgi?id=167668

Tim Jackson reported[9] that plague-client[10] was acting up and Michael Schwendt quickly provided[11] a patch which removed an assumption about how many bytes would be in the certificate. Dennis Gilmore commented "it's probably due to the new ca cert being 8096 bit and user certs are now all 2048 bit" and Chris Weyl filed[12] a bugzilla.

[9] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00993.html

[10] plague is the distributed package build system used by the EPEL repository

[11] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00996.html

[12] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01019.html

Later Paul Howarth encountered[13] what seemed to be a problem with his key or koji but which turned out, as suggested[14] by Jason Tibbitts to be due to denyhosts blocking him.

[13] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00970.html

[14] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00971.html

System Autodeath

Seth Vidal raised[1] the possibility of including a non-default option to include an "autodie" feature in future Fedora releases. The idea, originally expressed[2] in Glen Turner's blog is that each OS release should ship with an expected expiry date and a means to automatically remove the default route from that machine once the expiry date arrives. This would prevent old, unmaintained machines from being quietly exploited.

[1] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00853.html

[2] http://www.gdt.id.au/~gdt/blog/linux/autodeath.1024px

Agreement was expressed[3] by Jon Masters that it would be useful if "a sysadmin consciously wants to remember to remove a system from production/upgrade it after a certain time but then loses track of it[.]" Rahul Sundaram also thought[4] that something should be done but preferred the idea of modifying PackageKit to notify users when an upgrade was due and fixing up PreUpgrade to allow users to easily update without burning media. Jon Ciesla and Colin Walters wrangled over whether LiveUpgrade or PreUpgrade was the appropriate place to present such notifications with Colin disliking the LiveUpgrade due to support logistics. Richard Hughes was pleased to relate[5] that most of Rahul's desired feature had been implemented only a couple of days previously.

[3] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00855.html

[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00856.html

[5] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00932.html

Although Dave Jones was not worried[6] about desktop machines being abandoned this was contested. Dominik Mierzejewski related[7] anecdotes of people still running Fedora Core 2 while Stephen Smoogen regaled[8] the list with tales of hundreds of ancient Windows 3.11 clients on his network. Seth Vidal shared[9] Dave's insouciance and was more concerned about servers and "appliance-like machines" but promised to "look at putting a simple cron job together in a package to do this" while inviting anyone more motivated to go ahead and implement it.

[6] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00861.html

[7] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00862.html

[8] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00865.html

[9] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00863.html

A slight misunderstanding of Seth's intentions led[10] Horst H. von Brand to put the case "[...] against forcing people forward, even for their own good[.]" Horst argued that some systems could not be updated due to reliance on unmaintained legacy software. After Seth explained that he was not recommending that the "autodeath" feature be made a default Horst still expressed[11] a worry that casual installation followed by forgetfulness could result in the unexpected deactivation of systems later on. He suggested that instead of removing the default route "to a nag screen when EOL nears, offer to set up for upgrade, show (current) pointers to scripts helping check if 3rd party stuff will still work, ... install /that/ by default, allow to disable/uninstall it (even while it is nagging)." Seth objected[12] forcefully to describing the removal of the default route as "kills itself" and this resulted in a spate of alternate name suggestions.

[10] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00903.html

[11] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00929.html

[12] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00930.html

James Hubbard saw[13] similarities to "Windows Genuine Advantage where you can't use your machine anymore if you can't validate your installation" and suggested instead that users be notified of the EOL and in a separate part of the thread Jef Spaleta suggested that "autoannoy", via motd or the login banner, instead of "autodeath" might educate and help users more than cutting off connectivity.

[13] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00866.html

Commenting on responses from Matt Miller[14][15] and Jason Tibbitts[16], among others, Seth Vidalcommented[17] that it appeared that "all the .edu people seem to get this". But Horst was skeptical[18] that it was necessary to force sysadmins to make such changes.

[14] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00911.html

[15] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00935.html

[16] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00868.html

[17] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00869.html

[18] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00931.html

James Hubbard also made[19] a strong argument that lack of updates was as much of a security problem as being EOL'ed and thus any such measures should also be applied to systems lagging in their updates.

[19] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00959.html

GNUstep Filesystem Layout Discussion

A very clearly presented[1] explanation by Michel Salim kick started a discussion about how the GNUstep application development framework could finally be included in Fedora. Michel explained that GNUstep's idiosyncratic filesystem layout had previously made it impossible to install on an FHS[2] compliant system but that it was now possible. A choice had to be made for the layout of what GNUstep terms "application bundles" bearing in mind that "flattened" layouts are platform specific while "unflattened" can support multilib with little pain. Michel saw three main choices including a non-FHS-compliant one and noted that there were potential conflicts between utill-linux-ng and the default installation of GNUstep tools in /usr/bin/<arch>. He also noted that Debian had chosen to use an unflattened layout.

[1] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01007.html

[2] http://www.pathname.com/fhs/pub/fhs-2.3.html#INTRODUCTION

From this point onwards the discussion became a little difficult to follow due to the use of GNUstep specific terminology. The simplest information your correspondent could find was the online version[3] of the library-combo manpage and is probably essential reading before even attempting to grasp the outlines of the following.

[3] http://linux.die.net/man/7/library-combo

A suggestion[4] from Axel Thimm was "[...] to place [the GNUstep tools] directly under %{_bindir} and let rpm deal with the different archs as it does for all other %{_bindir} mixing of subarchs with colors etc" and to put the libraries under %{_libdir}. He argued that multilib or multiarch binaries were not generally supported in Fedora. Axel was also encouraging about the idea of starting a wiki page to attract as wide a possible contribution from GNUstep developers including non-Fedora contributors. Even more importantly Axel contrasted the ability of an unflattened layout to support different compilers and libraries to that of a flattened layout which could make OpenGroupware[5] conflict with other applications due to its use of libFoundation[6.] Kevin Koffler was unimpressed[7] with "packages which think they are a distro" and posited the solution that "[...] they need to be fixed to work with the system libraries instead (or the system libraries fixed to work with the packages[.]" While Axel agreed he explained[8] that what was being chosen was an "Objective-C runtime/ foundation library/graphical interface tuple (flattened)" and that it was necessary to allow a choice at runtime of the middle component in order to support applications depending on either libFoundation or gnustep-base[9]. Axel concluded "[...] IMHO we need to start with gnustep-make's FHS and non-flattened layout and fix it where it still needs fixing (gnustep-make FHS layout is very young and one could say that we are shaking out the bugs and when we are finished hopefully upstream will be glad to accept any patch we will have made to the FHS mode layout of gnustep-make)."

[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01009.html

[5] A groupware server integrating office suites through XML-based interfaces: http://www.opengroupware.org/en/about/mission.html

[6] Cocoa, libFoundation and gnustep-base all provide implementations of, and non-standard extensions to, the OpenStep API. Apart from licensing differences gnustep-base also has better platform coverage (Windows is not supported by the others) and full unicode support.

[7] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01021.html

[8] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01029.html

[9] http://www.gnustep.org/developers/suite.html

Discussion with Dominik 'Rathann' Mierzejewski and Kevin led Michel to post[10] that "[...] the consensus so far seems to be for using a flattened layout. Removing --disable-flattened from gnustep-make actually causes a much tighter adherence to the FHS, with %{_bindir} and %{_libdir} not containing any subdirectories." Michel was worried that this would result in duplicating data files and that "[...] keeping the unflattened layout might be too much trouble; if we are already flattening /usr/bin and /usr/lib*, might as well stick with a flattened layout after all."

[10] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01024.html

Apparently different conclusion were being reached at this stage and Axel attempted[11] to expand upon what he had said, explaining that this would result in having to choose between the Foundation libraries at buildtime. He presented unflattened layouts as allowing a choice of "libcombo" at runtime as opposed to flattened which forced a choice at buildtime. Dominik was[12] apparently comfortable with the idea of choosing between the two Foundation libraries and cited the precedent of not mixing lesstif and openmotif. To solve the problem he suggested "[put binaries in unflattened %{_libdir}/GNUstep/* and symlink to /usr/bin ." After Axel replied that the problem was the incompatible API/ABI of the Foundation libraries Michel posted[13] another summary of the current apparent state of knowledge.

[11] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01039.html

[12] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01040.html

[13] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01054.html

Varnish 2.0 Test Suite Fails in Rpmbuild

An interesting post from Ingvar Hagelund, the maintainer of the varnish http accelerator package, asked[1] why a test suite in the package would behave differently within the rpmbuild created environment than when run from an interactive shell. Ingvar had eliminated selinux as a possibility and speculated "[...] the problem seems to be related to some missing communication between the test scripts and the test server process."

[1] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00944.html

A response from Mogens Kjaer contained[2] a report that it was to do with a missing libvarnish.so.0 and wondered "[...] is the build system using an already installed libvarnish.so.0 if one is available and not the newly built libvarnish.so.0?" Jason Tibbitts suggested[3] that it was usual to reference such newly built libraries by manipulating LD_LIBRARY_PATH in the specfile. After Mogens added LD_LIBRARY_PATH and rebuilt from scratch he found[4] that all the tests were passed but that no varnish-libs had been installed and this led him to conclude that there was indeed a difference to the rpmbuild environment.

[2] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00945.html

[3] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00946.html

[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00948.html

Ingvar ended up[5] filing a bug report upstream with the conclusion that the soname version should be bumped as the old libraries were incompatible with varnish-2.0.

[5] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00951.html