Where's the Postmortem?Curated incident stories from the best engineering teams.https://wheresthepostmortem.com/When Someone Else Hijacked 1.1.1.1https://wheresthepostmortem.com/postmortems/cloudflare-1111-bgp-hijack-2024/https://wheresthepostmortem.com/postmortems/cloudflare-1111-bgp-hijack-2024/A Brazilian ISP announced a more-specific route for 1.1.1.1, hijacking DNS traffic across 300 networks in 70 countries.Thu, 27 Jun 2024 00:00:00 GMTWhen a Stolen Token in an Archived Repo Compromised an Entire Platformhttps://wheresthepostmortem.com/postmortems/heroku-oauth-token-breach-2022/https://wheresthepostmortem.com/postmortems/heroku-oauth-token-breach-2022/An attacker found a machine account token in an archived private repo, used it to steal customer OAuth tokens, and accessed private repositories across dozens of organizations.Thu, 07 Apr 2022 00:00:00 GMT73 Hours Down: When Service Discovery Took Down an Entire Platformhttps://wheresthepostmortem.com/postmortems/roblox-73-hour-outage-2021/https://wheresthepostmortem.com/postmortems/roblox-73-hour-outage-2021/A Consul upgrade and a BoltDB storage pathology combined to take Roblox offline for 73 hours.Thu, 28 Oct 2021 00:00:00 GMTThe Day Facebook Disappeared from the Internethttps://wheresthepostmortem.com/postmortems/facebook-global-outage-2021/https://wheresthepostmortem.com/postmortems/facebook-global-outage-2021/A backbone configuration change accidentally disconnected all Facebook data centers from each other and the internet for six hours.Mon, 04 Oct 2021 00:00:00 GMTOne Customer Setting Broke the Internet for an Hourhttps://wheresthepostmortem.com/postmortems/fastly-global-cdn-outage-2021/https://wheresthepostmortem.com/postmortems/fastly-global-cdn-outage-2021/A single valid customer config change triggered a dormant bug that took down 85% of Fastly's CDN, knocking Amazon, Reddit, and the BBC offline.Tue, 08 Jun 2021 00:00:00 GMTWhen Everyone Came Back from Holiday and Slack Couldn't Handle Ithttps://wheresthepostmortem.com/postmortems/slack-first-monday-outage-2021/https://wheresthepostmortem.com/postmortems/slack-first-monday-outage-2021/Cold client caches after the holiday break overwhelmed AWS Transit Gateways, triggering auto-scaling chaos that made the outage worse.Mon, 04 Jan 2021 00:00:00 GMT43 Seconds of Network Partition, 24 Hours of Degraded Servicehttps://wheresthepostmortem.com/postmortems/github-database-incident-2018/https://wheresthepostmortem.com/postmortems/github-database-incident-2018/A 43-second network partition triggered an automated cross-region MySQL failover that created a split-brain, degrading GitHub for 24 hours.Sun, 21 Oct 2018 00:00:00 GMTA Typo That Took Down the Internet for Four Hourshttps://wheresthepostmortem.com/postmortems/aws-s3-outage-2017/https://wheresthepostmortem.com/postmortems/aws-s3-outage-2017/An engineer mistyped a command while debugging, removing far more S3 servers than intended. The restart took four hours because S3 had never been fully restarted.Tue, 28 Feb 2017 00:00:00 GMTWhen a Database Engineer Accidentally Deleted the Production Databasehttps://wheresthepostmortem.com/postmortems/gitlab-database-deletion-2017/https://wheresthepostmortem.com/postmortems/gitlab-database-deletion-2017/An engineer ran rm -rf on production. Five backup strategies all failed.Tue, 31 Jan 2017 00:00:00 GMTHow Dead Code Caused a $440 Million Loss in 45 Minuteshttps://wheresthepostmortem.com/postmortems/knight-capital-440m-trading-loss-2012/https://wheresthepostmortem.com/postmortems/knight-capital-440m-trading-loss-2012/A botched deployment activated 9-year-old dead code that bought high and sold low for 45 minutes straight.Wed, 01 Aug 2012 00:00:00 GMT