From Fedora Project Wiki

< FWN

(add some links to the previous high-profile project problems, add pubilc reaction, add obvious signs of progress)
Line 5: Line 5:
[1] http://fedoraproject.org/wiki/FWN/LatestIssue#General_Outage_of_Fedora_Infrastructure
[1] http://fedoraproject.org/wiki/FWN/LatestIssue#General_Outage_of_Fedora_Infrastructure


An update[2] was posted by [[PaulFrields|Paul Frields]] on 18 August which states:
An update[2] was posted by [[PaulFrields|Paul Frields]] on 18 August which listed the services which had returned to normal and were expected to return to normal soon. Public speculation latched on[3][4] to the fact that the SSH keys of "fedorahosted" had changed. Most guesses used this as evidence that something similar to the recent 2008 Debian OpenSSL vulnerabilities (not be confused with the 2003 Debian Project compromise[5] which was due to a 0-day kernel exploit or the 2006 compromise[6]) had occurred. FAS holders received an email asking them to reset their passwords as a precautionary measure which further heightened suspicions that something similar to the SSL problem had occurred.


"Our team has been hard at work for several days now, restoring services
[2] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00011.html
in the Fedora infrastructure. We started with what we identified as
Fedora's "critical path," those systems required to restore minimum
daily operation. That work to be completely finished by the end of the
day. We then move on to our other value services to complete them as
soon as possible.


Please give the infrastructure team the time they need to do this
[3] http://lwn.net/Articles/294547/
demanding work. They have been doing a spectacular job and deserve the
absolute highest credit.


The systems that are now back online and usable include the following:
[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00790.html
* Puppet, Xen and FAS hosts
* app1, app3, and app4
* database and proxy servers
* the majority of the Xen guest machines
* serverbeach5, serverbeach4
* Fedora Hosted**


The systems that should be available very soon:
[5] A key developer's machine was compromised due to a kernel exploit[5a] and then password sniffers were installed which provided the attacker(s) with root access to at least one key Debian server (klecker.debian.org). This was used as a staging post to install another sniffer and a chain of two more servers were compromised. As as result of modifications made to the one of the kernels it started OOPsing and investigations of this revealed the problem.
* asterisk1 and collab1
* cvs1
* builders, x86 and ppc
* Fedora People


We know the community is awaiting more detail on the past week's
[5a] http://www.securityfocus.com/archive/1/346095/2003-11-28/2003-12-04/0
activities and their causes. We're preparing a timeline and details and
will make them available in the near future. We appreciate the
community's patience, and will continue to post updates to the
fedora-announce-list as soon as possible."


= = =
[6] http://www.debian.org/News/2006/20060713
** New SSH fingerprint for Fedora Hosted:
  e6:b3:68:51:98:2d:4c:dc:63:27:46:65:51:d5:f0:7a
''


[2] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00011.html
[7] Metasploit has an excellent writeup on the topic here: http://www.metasploit.com/users/hdm/tools/debian-openssl/

Revision as of 01:18, 20 August 2008

Breaking News on the Infrastructure Outage

This special issue of FWN will be updated to reflect new information as soon as it is made available to us. Currently we are still relying on the information posted by Paul Frields on @fedora-infrastructure just like the rest of you. As reported in FWN#139 "General Outage of Fedora Infrastructure" all that is known is that the problems became obvious to a wide audience on August 13th.

[1] http://fedoraproject.org/wiki/FWN/LatestIssue#General_Outage_of_Fedora_Infrastructure

An update[2] was posted by Paul Frields on 18 August which listed the services which had returned to normal and were expected to return to normal soon. Public speculation latched on[3][4] to the fact that the SSH keys of "fedorahosted" had changed. Most guesses used this as evidence that something similar to the recent 2008 Debian OpenSSL vulnerabilities (not be confused with the 2003 Debian Project compromise[5] which was due to a 0-day kernel exploit or the 2006 compromise[6]) had occurred. FAS holders received an email asking them to reset their passwords as a precautionary measure which further heightened suspicions that something similar to the SSL problem had occurred.

[2] https://www.redhat.com/archives/fedora-announce-list/2008-August/msg00011.html

[3] http://lwn.net/Articles/294547/

[4] https://www.redhat.com/archives/fedora-devel-list/2008-August/msg00790.html

[5] A key developer's machine was compromised due to a kernel exploit[5a] and then password sniffers were installed which provided the attacker(s) with root access to at least one key Debian server (klecker.debian.org). This was used as a staging post to install another sniffer and a chain of two more servers were compromised. As as result of modifications made to the one of the kernels it started OOPsing and investigations of this revealed the problem.

[5a] http://www.securityfocus.com/archive/1/346095/2003-11-28/2003-12-04/0

[6] http://www.debian.org/News/2006/20060713

[7] Metasploit has an excellent writeup on the topic here: http://www.metasploit.com/users/hdm/tools/debian-openssl/