FWN/Issue140

Breaking News on the Infrastructure Outage

This special issue of FWN covers the infrastructure problems as reported in FWN#139 "General Outage of Fedora Infrastructure" all that was known[1] was that the problems became obvious to a wide audience on August 13th.

[1] http://fedoraproject.org/wiki/FWN/LatestIssue#General_Outage_of_Fedora_Infrastructure

An update[2] was posted by Paul Frields on 18 August which listed the services which had returned to normal and were expected to return to normal soon. Public speculation latched on[3][4] to the fact that the SSH keys of "fedorahosted" had changed. Most guesses used this as evidence that something similar to the recent 2008 Debian OpenSSL vulnerabilities (not be confused with the 2003 Debian Project compromise[5] which was due to a 0-day kernel exploit or the 2006 compromise[6]) had occurred. FAS holders received an email asking them to reset their passwords as a precautionary measure which further heightened suspicions that something similar to the SSL problem had occurred. Some confusion prevailed[8] on @fedora-devel as to whether it was possible to trust the new key fingerprint on the website. JimMeyering added[8a] a useful post which explained how to change from using a DSA ssh key to an RSA ssh key. Overall there was a surprisingly low level of public discussion of the problem and it was not until 18 August that some complaints about the lack of information were expressed[9] on @fedora-list.

[2] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00011.html

[3] http://lwn.net/Articles/294547/

[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00790.html

[5] A key developer's machine was compromised due to a kernel exploit[5a] and then password sniffers were installed which provided the attacker(s) with root access to at least one key Debian server (klecker.debian.org). This was used as a staging post to install another sniffer and a chain of two more servers were compromised. As as result of modifications made to the one of the kernels it started OOPsing and investigations of this revealed the problem.

[5a] http://www.securityfocus.com/archive/1/346095/2003-11-28/2003-12-04/0

[6] http://www.debian.org/News/2006/20060713

[7] Metasploit has an excellent writeup on the topic here: http://www.metasploit.com/users/hdm/tools/debian-openssl/

[8] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00841.html

[9] https://www.redhat.com/archives/fedora-list/2008-August/msg01953.html

On 22 August Paul Frields posted[10] an announcement that stated that an intrusion had been detected on Fedora Project machines including the package signing machine. An extensive audit suggests that the passphrase to the key was not sniffed yet it was decided to treat the intrusion as having potentially compromised the key. The result of this has been that all the keys are being reissued.

Date Fri, 22 Aug 2008 8:00 AM ( 9 mins 59 secs ago ) Text view Print view From "Paul W. Frields" <stickster@gmail.com> To "fedora-announce-list" <fedora-announce-list@redhat.com> [Add] Subject Infrastructure report, 2008-08-22 UTC 1200 Show full header Last week we discovered that some Fedora servers were illegally accessed. The intrusion into the servers was quickly discovered, and the servers were taken offline.

Security specialists and administrators have been working since then to analyze the intrusion and the extent of the compromise as well as reinstall Fedora systems. We are using the requisite outages as an opportunity to do other upgrades for the sake of functionality as well as security. Work is ongoing, so please be patient. Anyone with pertinent information relating to this event is asked to contact fedora-legal@redhat.com.

One of the compromised Fedora servers was a system used for signing Fedora packages. However, based on our efforts, we have high confidence that the intruder was not able to capture the passphrase used to secure the Fedora package signing key. Based on our review to date, the passphrase was not used during the time of the intrusion on the system and the passphrase is not stored on any of the Fedora servers.

While there is no definitive evidence that the Fedora key has been compromised, because Fedora packages are distributed via multiple third-party mirrors and repositories, we have decided to convert to new Fedora signing keys. This may require affirmative steps from every Fedora system owner or administrator. We will widely and clearly communicate any such steps to help users when available.

Among our other analyses, we have also done numerous checks of the Fedora package collection, and a significant amount of source verification as well, and have found no discrepancies that would indicate any loss of package integrity. These efforts have also not resulted in the discovery of additional security vulnerabilities in packages provided by Fedora.

Our previous warnings against further package updates were based on an abundance of caution, out of respect for our users. This is also why we are proceeding with plans to change the Fedora package signing key. We have already started planning and implementing other additional safeguards for the future. At this time we are confident there is little risk to Fedora users who wish to install or upgrade signed Fedora packages.

In connection with these events, Red Hat, Inc. detected an intrusion of certain of its computer systems and has issued a communication to Red Hat Enterprise Linux users which can be found at http://rhn.redhat.com/errata/RHSA-2008-0855.html. This communication states in part, "Last week Red Hat detected an intrusion on certain of its computer systems and took immediate action. While the investigation into the intrusion is on-going, our initial focus was to review and test the distribution channel we use with our customers, Red Hat Network (RHN) and its associated security measures. Based on these efforts, we remain highly confident that our systems and processes prevented the intrusion from compromising RHN or the content distributed via RHN and accordingly believe that customers who keep their systems updated using Red Hat Network are not at risk. We are issuing this alert primarily for those who may obtain Red Hat binary packages via channels other than those of official Red Hat subscribers."

It is important to note that the effects of the intrusion on Fedora and Red Hat are *not* the same. Accordingly, the Fedora package signing key is not connected to, and is different from, the one used to sign Red Hat Enterprise Linux packages. Furthermore, the Fedora package signing key is also not connected to, and is different from, the one used to sign community Extra Packages for Enterprise Linux (EPEL) packages.

We will continue to keep the Fedora community notified of any updates.

Thank you again for your patience.