From Fedora Project Wiki

< FWN‎ | Beats

(→‎Mysterious Fedora Compromise: correct BrunoWolff name as requested)
(1/5 of @fedora-devel, needs lots of love)
Line 7: Line 7:
Contributing Writer: [[OisinFeeley|Oisin Feeley]]
Contributing Writer: [[OisinFeeley|Oisin Feeley]]


=== Mysterious Fedora Compromise ===
=== Intrusion Recovery Slow and Steady ===


The mysterious unavailability of much of the FedoraProject infrastructure (see FWN#139 "General Outage of Fedora Infrastructure"[1]) continued to provoke speculation during the week. Some light was shed[2] on 22-08-2008 when [[PaulFrields|Paul Frields]] announced that there had been an intrusion onto several FedoraProject servers. The announcement emphasized that although one of the servers was used for signing rpm packages it was believed, based upon an absence of positive evidence, that the key and packages had not been tampered with. Nevertheless prudence and caution were being exercised and the opportunity was being seized to completely re-install and upgrade all systems. As a result it was necessary for most contributors to reset authentication tokens of various types (see this same issue[3].) It also appeared[4] that a concurrent event had led to the signing of some Red Hat OpenSSH packages, but that these had been quickly detected and had not led to the distribution of compromised packages.
A politely phrased request[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01102.html ] was made on 25-08-2008 by [[MikeChambers|Mike Chambers]] for information about when normal service would resume. Enigmatically [[DominikMierzejewski|Dominik 'Rathann' Mierzejewski]] observed[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01122.html ] that there had been “some speculation on fedora-advisory-board that might explain the information blackout, so please don't jump to conclusions until you really know what happened” This led [[ChrisAdams|Chris Adams]] to observe that the list archives appeared to be offline and to restate the request for information “[...] in the absence of information, rumors and speculation fill the gap (which is not good).
 
[1] http://fedoraproject.org/wiki/FWN/Issue139#General_Outage_of_Fedora_Infrastructure
Several days later (on 28-08-2008) a similar request was made[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01308.html ] by [[AlanDunn|Alan Dunn]]. He wondered whether ''bodhi'' was pushing updates out again and JoshBoyer responded[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01309.html ] that planning and implementation of “how to revoke the current gpg key used to sign RPMs” were in progress. [[JesseKeating|Jesse Keating]] cautioned[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01310.html ] that the migration to a new key would be slow “I'm currently re-signing all of the 8 and 9 content with these new keys so that we can make them available along with the new updates with the new key for these product lines. This is going to take some time due to the nature of how our signing works.” A proposal mooted[footnote: http://lists.fedoraproject.org/pipermail/rel-eng/2008-August/001627.html ] on @rel-eng by [[WarrenTogami|Warren Togami]] provided some insight into at least the part of the plans that involve the problem of how to distribute a new package signing key.
 
 
[2] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00012.html
''nodata'' asked[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01313.html ] whether the new plans included a means to push out critical security updates even while there was a general outage. The thinking behind this seems to be that an attacker could decide to knock out Fedora infrastructure in order to gain some time to exploit a known vulnerability even if a simple fix existed.  [[JesseKeating|Jesse Keating]] replied[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01314.html ] confidently that in such a scenario the Fedora Project would do “whatever it takes [...] to get a critical update onto a public webserver should the need arise” and cautioned against trying to plan for every possible scenario. [[ToshioKuratomi|Toshio Kuratomi]] added[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01316.html ] that although it might be possible to speed up recovery “[...] unfortunately if the infrastructure problem is bad enough, there's no way we can push package X out until the problem is at least partially resolved.
 
[3] "Epiphany, Konqueror and Plague Hamper Updating SSL Client-side Certificates"
On 27-08-2008 [[PaulJohnson|Paul Johnson]] noted that it was possible to “compose and build” and asked “when will updates via yum become available for rawhide?” [[JeremyKatz|Jeremy Katz]] responded[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01249.html ] that “[a]t the moment, the compose is falling over for new reasons unrelated to the infrastructure changes. Hopefully we'll see a rawhide make its way out to the masses real soon now.” Later [[MikeChambers|Mike Chambers]] and [[OlaThoresen|Ola Thoresen]] reported[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01350.html ] that updating from Fedora 9 to Rawhide seemed to be workingSeveral Rawhide Reports appeared[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01339.html ].
[4] http://www.redhat.com/security/data/openssh-blacklist.html
 
Prior to that announcement all that was known was that there were problems on the build servers which became obvious to a wide audience on 13-08-2008 and that users were advised[5] on 14-08-2008 not to install updates. The wiki and email lists continued to function during this period thanks to the efforts of their administrators.
 
[5] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00008.html
 
An update[6] was posted on 18-08-2008 by [[PaulFrields|Paul Frields]] that listed the services which had returned to normal and those which were expected to return to normal soon. Public speculation latched on[7][8] to the changing of "fedorahosted" SSH keys. Most guesses used this as evidence that something similar to the recent 2008 Debian OpenSSL vulnerabilities[9] had occurred. Some confusion prevailed[10] on @fedora-devel as to whether it was possible to trust the new key fingerprint on the website. [[JimMeyering|Jim Meyering]] added[11] a useful post which explained how to change from using a DSA ssh key to an RSA ssh key. Overall there was a surprisingly low level of public discussion of the problem and it was not until 18-08-2008 that some complaints about the lack of information were expressed[12] on @fedora-list. On 22-08-2008 another bulletin was released[13] by [[PaulFrields|Paul Frields]] that explained that the Fedora key had not been obviously compromised but that it was still being changed on the precautionary principle.
 
[6] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00011.html
 
[7] http://lwn.net/Articles/294547/
 
[8] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00790.html
 
[9] Metasploit has an excellent writeup on the topic here: http://www.metasploit.com/users/hdm/tools/debian-openssl/
 
[10] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00841.html
 
[11] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00845.html
 
[12] https://www.redhat.com/archives/fedora-list/2008-August/msg01953.html
 
[13] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00012.html
 
Although many responses to complaints about the limited information suggested[14] that the Fedora developers could be trusted to "do the right thing" in terms of alerting users to immediate threats of compromise there was still strong disquiet expressed[15] over the lack of information. This also occurred[16] on @fedora-list. [[JeffSpaleta|Jef Spaleta]] wondered[17] if there might be a better way of getting information out than relying on everyone to subscribe to @fedora-announce. [[AlanCox|Alan Cox]] suggested[18] that an RSS feed for critical announcements could be picked up by a system's default package updater and that repositories could communicate errors to ''yum''. Alan was also unhappy with the absence of important notices on the very front of the website and as a separate matter criticized the content of the information issued: "[...] leaving people in the dark assuming the worst [is] a very bad way to create long term trust." [[BrunoWolff|Bruno Wolff III]] also suggested[19] that information should have been conveyed by a more central announcement on fedoraproject.org and co-ordination with media such as LWN.net.
 
[14] https://www.redhat.com/archives/fedora-list/2008-August/msg01955.html
 
[15] https://www.redhat.com/archives/fedora-list/2008-August/msg02034.html
 
[16] https://www.redhat.com/archives/fedora-list/2008-August/msg02426.html
 
[17] https://www.redhat.com/archives/fedora-list/2008-August/msg02010.html
 
[18] https://www.redhat.com/archives/fedora-list/2008-August/msg02012.html
 
[19] https://www.redhat.com/archives/fedora-list/2008-August/msg02013.html
 
Most comments on @fedora-devel made a point of thanking the Fedora Infrastructure admins, even to the extent of providing internet beers[20] and [[HansdeGoede|Hans de Goede]] commented[21] that "Even before the unfortunate events of the last few weeks the infrastructure team has been doing an amazing job[.]"
 
[20] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01037.html
 
[21] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01028.html
 
=== Epiphany, Konqueror and Plague Hamper Updating SSL Client-side Certificates ===
 
As part of the general overhaul of Fedora Project infrastructure security occasioned by the recent intrusion[1] [[DennisGilmore|Dennis Gilmore]] advised[2] that the certificate authority governing access to cvs.fedoraproject.org and koji.fedoraproject.org had been changed. As a result it was necessary for those who wished to build packages or upload to the lookaside cache to manually import a new client-side certificate and to remove their old certificate.
 
[1] http://fedoraproject.org/wiki/FWN/Issue139#General_Outage_of_Fedora_Infrastructure
 
[2] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00962.html
 
[[MartinSourada|Martin Sourada]] reported[3] that after following the procedure he still received a <code>(Error code: ssl_error_unknown_ca_alert)</code>. [[KaiEngert|Kai Engert]] suggested[4] that Martin might need to import the Fedora Project root CA certificate after verifying its fingerprint. As Martin had allowed exceptions for the Fedora websites this was not the problem and it turned out[5] to be due to a problem with the ''epiphany'' web browser failing to offer an option to remove old certificates.
 
[3] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00978.html
 
[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00982.html
 
[5] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00989.html
 
Problems were also experienced[6] by [[JoseMatos|Jose Matos]], but this time with the ''konqueror'' web browser (version 4.1.0). [[KevinKoffler|Kevin Koffler]] replied[7] that in ''KDE 4'' there was no support for SSL certificate authentication with ''konqueror'' and pasted a link[8] to an upstream bug report filed with the KDE project on this issue.
 
[6] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00983.html
 
[7] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00999.html
 
[8] https://bugs.kde.org/show.bug.cgi?id=167668
 
[[TimJackson|Tim Jackson]] reported[9] that ''plague-client''[10] was acting up and [[MichaelSchwendt|Michael Schwendt]] quickly provided[11] a patch which removed an assumption about how many bytes would be in the certificate. [[DennisGilmore|Dennis Gilmore]] commented "it's probably due to the new ca cert being 8096 bit and user certs are now all 2048 bit" and [[ChrisWeyl|Chris Weyl]] filed[12] a bugzilla.
 
[9] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00993.html
 
[10] ''plague'' is the distributed package build system used by the EPEL repository
 
[11] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00996.html
 
[12] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01019.html
 
Later [[PaulHowarth|Paul Howarth]] encountered[13] what seemed to be a problem with his key or ''koji'' but which turned out, as suggested[14] by [[JasonTibbitts|Jason Tibbitts]] to be due to ''denyhosts'' blocking him.
 
[13] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00970.html
 
[14] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00971.html
 
=== System Autodeath ===
 
[[SethVidal|Seth Vidal]] raised[1] the possibility of including a non-default option to include an "autodie" feature in future Fedora releases. The idea, originally expressed[2] in [[GlenTurner|Glen Turner's]] blog is that each OS release should ship with an expected expiry date and a means to automatically remove the default route from that machine once the expiry date arrives. This would prevent old, unmaintained machines from being quietly exploited.
 
[1] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00853.html
 
[2] http://www.gdt.id.au/~gdt/blog/linux/autodeath.1024px
 
Agreement was expressed[3] by [[JonMasters|Jon Masters]] that it would be useful if "a sysadmin consciously wants to remember to remove a system from production/upgrade it after a certain time but then loses track of it[.]" [[RahulSundaram|Rahul Sundaram]] also thought[4] that something should be done but preferred the idea of modifying PackageKit to notify users when an upgrade was due and fixing up [[PreUpgrade]] to allow users to easily update without burning media. [[JonCiesla|Jon Ciesla]] and [[ColinWalters|Colin Walters]] wrangled over whether [[SIGs/LiveUpgrade|LiveUpgrade]] or [[Anaconda/Features/PreUpgrade|PreUpgrade]] was the appropriate place to present such notifications with Colin disliking the LiveUpgrade due to support logistics. [[RichardHughes|Richard Hughes]] was pleased to relate[5] that most of Rahul's desired feature had been implemented only a couple of days previously. 
 
[3] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00855.html
 
[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00856.html
 
[5] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00932.html
 
Although [[DaveJones|Dave Jones]] was not worried[6] about desktop machines being abandoned this was contested. [[DominikMierzejewski|Dominik Mierzejewski]] related[7] anecdotes of people still running Fedora Core 2 while [[StephenSmoogen|Stephen Smoogen]] regaled[8] the list with tales of hundreds of ancient Windows 3.11 clients on his network. [[SethVidal|Seth Vidal]] shared[9] Dave's insouciance and was more concerned about servers and "appliance-like machines" but promised to "look at putting a simple cron job together in a package to do this" while inviting anyone more motivated to go ahead and implement it.
 
[6] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00861.html
 
[7] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00862.html
 
[8] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00865.html
 
[9] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00863.html
 
A slight misunderstanding of Seth's intentions led[10] [[HorstvonBrand|Horst H. von Brand]] to put the case "[...] against forcing people forward, even for their own good[.]" Horst argued that some systems could not be updated due to reliance on unmaintained legacy software. After Seth explained that he was not recommending that the "autodeath" feature be made a default Horst still expressed[11] a worry that casual installation followed by forgetfulness could result in the unexpected deactivation of systems later on. He suggested that instead of removing the default route "to a nag screen when EOL nears, offer to set up for upgrade, show (current) pointers to scripts helping check if 3rd party stuff will still work, ... install /that/ by default, allow to disable/uninstall it (even while it is nagging)." Seth objected[12] forcefully to describing the removal of the default route as "kills itself" and this resulted in a spate of alternate name suggestions.
 
[10] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00903.html
 
[11] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00929.html
 
[12] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00930.html
 
[[JamesHubbard|James Hubbard]] saw[13] similarities to "Windows Genuine Advantage where you can't use your machine anymore if you can't validate your installation" and suggested instead that users be notified of the EOL and in a separate part of the thread [[JefSpaleta|Jef Spaleta]] suggested that "autoannoy", via ''motd'' or the login banner, instead of "autodeath" might educate and help users more than cutting off connectivity.
 
[13] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00866.html
 
Commenting on responses from [[MattMiller|Matt Miller]][14][15] and [[JasonTibbitts|Jason Tibbitts]][16], among others, [[SethVidal|Seth Vidal]]commented[17] that it appeared that "all the .edu people seem to get this". But Horst was skeptical[18] that it was necessary to force sysadmins to make such changes.
 
[14] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00911.html
 
[15] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00935.html
 
[16] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00868.html
 
[17] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00869.html
 
[18] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00931.html
 
[[JamesHubbard|James Hubbard]] also made[19] a strong argument that lack of
updates was as much of a security problem as being EOL'ed and thus any such
measures should also be applied to systems lagging in their updates.
 
[19] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00959.html
 
=== GNUstep Filesystem Layout Discussion ===
 
A very clearly presented[1] explanation by [[MichelSalim|Michel Salim]] kick started a discussion about how the GNUstep application development framework could finally be included in Fedora. Michel explained that GNUstep's idiosyncratic filesystem layout had previously made it impossible to install on an FHS[2] compliant system but that it was now possible. A choice had to be made for the layout of what GNUstep terms "application bundles" bearing in mind that "flattened" layouts are platform specific while "unflattened" can support multilib with little pain. Michel saw three main choices including a non-FHS-compliant one and noted that there were potential conflicts between <code>utill-linux-ng</code> and the default installation of GNUstep tools in /usr/bin/<arch>. He also noted that Debian had chosen to use an unflattened layout.
 
[1] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01007.html
 
[2] http://www.pathname.com/fhs/pub/fhs-2.3.html#INTRODUCTION
 
From this point onwards the discussion became a little difficult to follow due to the use of GNUstep specific terminology. The simplest information your correspondent could find was the online version[3] of the <code>library-combo</code> manpage and is probably essential reading before even attempting to grasp the outlines of the following.
 
[3] http://linux.die.net/man/7/library-combo
 
A suggestion[4] from [[AxelThimm|Axel Thimm]] was "[...] to place [the GNUstep tools] directly under %{_bindir} and let rpm deal with the different archs as it does for all other %{_bindir} mixing of subarchs with colors etc" and to put the libraries under <code>%{_libdir}</code>. He argued that multilib or multiarch binaries were not generally supported in Fedora. Axel was also encouraging about the idea of starting a wiki page to attract as wide a possible contribution from GNUstep developers including non-Fedora contributors. Even more importantly Axel contrasted the ability of an unflattened layout to support different compilers and libraries to that of a flattened layout which could make OpenGroupware[5] conflict with other applications due to its use of <code>libFoundation</code>[6.] [[KevinKoffler|Kevin Koffler]] was unimpressed[7] with "packages which think they are a distro" and posited the solution that "[...] they need to be fixed to work with the system libraries instead (or the system libraries fixed to work with the packages[.]" While Axel agreed he explained[8] that what was being chosen was an "Objective-C runtime/ foundation library/graphical interface tuple (flattened)" and that it was necessary to allow a choice at runtime of the middle component in order to support applications depending on either libFoundation or gnustep-base[9]. Axel concluded "[...] IMHO we need to start with gnustep-make's FHS and non-flattened layout and fix it where it still needs fixing (gnustep-make FHS layout is very young and one could say that we are shaking out the bugs and when we are finished hopefully upstream will be glad to accept any patch we will have made to the FHS mode layout of gnustep-make)."
 
[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01009.html
 
[5] A groupware server integrating office suites through XML-based interfaces: http://www.opengroupware.org/en/about/mission.html
 
[6] Cocoa, libFoundation and gnustep-base all provide implementations of, and non-standard extensions to, the OpenStep API. Apart from licensing differences gnustep-base also has better platform coverage (Windows is not supported by the others) and full unicode support.
 
[7] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01021.html
 
[8] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01029.html
 
[9] http://www.gnustep.org/developers/suite.html
 
Discussion with [[DominikMierzejewski|Dominik 'Rathann' Mierzejewski]] and Kevin led Michel to post[10] that "[...] the consensus so far seems to be for using a flattened layout. Removing --disable-flattened from gnustep-make actually causes a much tighter adherence to the FHS, with %{_bindir} and %{_libdir} not containing any subdirectories." Michel was worried that this would result in duplicating data files and that "[...] keeping the unflattened layout might be too much trouble; if we are already flattening /usr/bin and /usr/lib*, might as well stick with a flattened layout after all."
 
[10] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01024.html
 
Apparently different conclusions were being reached at this stage and Axel attempted[11] to expand upon what he had said, explaining that this would result in having to choose between the Foundation libraries at buildtime. He presented unflattened layouts as allowing a choice of "libcombo" at runtime as opposed to flattened which forced a choice at buildtime. Dominik was[12] apparently comfortable with the idea of choosing between the two Foundation libraries and cited the precedent of not mixing ''lesstif'' and ''openmotif''. To solve the problem he suggested "[put binaries in unflattened %{_libdir}/GNUstep/* and symlink to /usr/bin ." After Axel replied that the problem was the incompatible API/ABI of the Foundation libraries Michel posted[13] another summary of the current apparent state of knowledge.
 
[11] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01039.html
 
[12] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01040.html
 
[13] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01054.html
 
=== Varnish 2.0 Test Suite Fails in Rpmbuild ===
 
An interesting post from [[IngvarHagelund|Ingvar Hagelund]], the maintainer of the ''varnish'' http accelerator package, asked[1] why a test suite in the package would behave differently within the ''rpmbuild'' created environment than when run from an interactive shell. Ingvar had eliminated ''selinux'' as a possibility and speculated "[...] the problem seems to be related to some missing communication between the test scripts and the test server process."
 
[1] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00944.html
 
A response from [[MogensKjaer|Mogens Kjaer]] contained[2] a report that it was to do with a missing <code>libvarnish.so.0</code> and wondered "[...] is the build system using an already installed libvarnish.so.0 if one is available and not the newly built libvarnish.so.0?" [[JasonTibbitts|Jason Tibbitts]] suggested[3] that it was usual to reference such newly built libraries by manipulating <code>LD_LIBRARY_PATH</code> in the specfile. After Mogens added LD_LIBRARY_PATH and rebuilt from scratch he found[4] that all the tests were passed but that no varnish-libs had been installed and this led him to conclude that there was indeed a difference to the rpmbuild environment.   
 
[2] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00945.html
 
[3] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00946.html
 
[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00948.html
 
Ingvar ended up[5] filing a bug report upstream with the conclusion that the soname version should be bumped as the old libraries were incompatible with varnish-2.0.
 
[5] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00951.html

Revision as of 18:42, 30 August 2008

Developments

In this section the people, personalities and debates on the @fedora-devel mailing list are summarized.

Contributing Writer: Oisin Feeley

Intrusion Recovery Slow and Steady

A politely phrased request[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01102.html ] was made on 25-08-2008 by Mike Chambers for information about when normal service would resume. Enigmatically Dominik 'Rathann' Mierzejewski observed[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01122.html ] that there had been “some speculation on fedora-advisory-board that might explain the information blackout, so please don't jump to conclusions until you really know what happened” This led Chris Adams to observe that the list archives appeared to be offline and to restate the request for information “[...] in the absence of information, rumors and speculation fill the gap (which is not good).”

Several days later (on 28-08-2008) a similar request was made[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01308.html ] by Alan Dunn. He wondered whether bodhi was pushing updates out again and JoshBoyer responded[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01309.html ] that planning and implementation of “how to revoke the current gpg key used to sign RPMs” were in progress. Jesse Keating cautioned[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01310.html ] that the migration to a new key would be slow “I'm currently re-signing all of the 8 and 9 content with these new keys so that we can make them available along with the new updates with the new key for these product lines. This is going to take some time due to the nature of how our signing works.” A proposal mooted[footnote: http://lists.fedoraproject.org/pipermail/rel-eng/2008-August/001627.html ] on @rel-eng by Warren Togami provided some insight into at least the part of the plans that involve the problem of how to distribute a new package signing key.

nodata asked[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01313.html ] whether the new plans included a means to push out critical security updates even while there was a general outage. The thinking behind this seems to be that an attacker could decide to knock out Fedora infrastructure in order to gain some time to exploit a known vulnerability even if a simple fix existed. Jesse Keating replied[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01314.html ] confidently that in such a scenario the Fedora Project would do “whatever it takes [...] to get a critical update onto a public webserver should the need arise” and cautioned against trying to plan for every possible scenario. Toshio Kuratomi added[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01316.html ] that although it might be possible to speed up recovery “[...] unfortunately if the infrastructure problem is bad enough, there's no way we can push package X out until the problem is at least partially resolved.”

On 27-08-2008 Paul Johnson noted that it was possible to “compose and build” and asked “when will updates via yum become available for rawhide?” Jeremy Katz responded[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01249.html ] that “[a]t the moment, the compose is falling over for new reasons unrelated to the infrastructure changes. Hopefully we'll see a rawhide make its way out to the masses real soon now.” Later Mike Chambers and Ola Thoresen reported[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01350.html ] that updating from Fedora 9 to Rawhide seemed to be working. Several Rawhide Reports appeared[footnote: https://www.redhat.com/archives/fedora-devel-list/2008-August/msg01339.html ].