What:: Everything broke
When: 13:13 PST to 15:43 PST
How: At 13:13 PST, Swamp Master @Aether initiated an planned upgrade of the site from version 0.21 to version 0.22. While the upgrade initially looked fine, he re-ran it because he realized he was using an older version of the upgrade script. After the upgrade was complete, the site appeared to be unreachable due to an invalid certificate error. Swamp Master @sWaMPqUeeN had just finished getting her nails done and they needed to pick up supplies at the cat shelter, so the site remained offline and inaccessible until they got back. At ~15:40 PST he realized that half of the containers weren't running, and re-issued the startup command. All services appear to be running fine now. We're on version 0.22, although the admin control panel still says 0.21.
We here at the Data Swamp are deeply sorry for this unplanned outage. We know the thousands of users of this site likely went into apoplectic shock upon realizing that the site was inaccessible. We promise that future upgrades will be more error-prone and will hopefully crash this site permanently so we can all be free of this indelible curse.
