From Fedora Project Wiki

Proxy01 down issue

Description

Proxy01 went down with lots of nagios notifications following. This also dragged down koji from internal (internally, koji.fp.o is only mapped to proxy01).


When the issue presented itself

February 18, 2017, 09:15 UTC was the first nagios notification.

When it recovered (or got fixed)

09:20 UTC, rerouted internal proxy01 to proxy10 and disabled proxy01 in DNS to prevent user issues.

09:40 UTC, after clearing koji access logs, proxy01 httpd started.

09:45 UTC, proxy01 re-enabled in DNS.

Root cause

Koji access logs filled up proxy01's drive to 100%. For some reason, logrotate had not rotated the 20170218 logs out to an xz-compressed form, meaning there were lots of multi-GB access files, filling up the disk entirely.


Service owners


Follow-up steps

  • Figure out why logrotate didn't work. => looks like it did work, but the files just growed so hard that it never got the chance to rotate.
  • Figure out why access logs were bigger than usual (access logs on proxy01 gone, will need to come from hubs).
  • Make internal use both proxy01 and proxy10 for koji access

Future ideas

  • Make it easier to disable proxy by only providing name in cmds/ files.