How 59 Million Deletes Broke GitHub: The MySQL Migration Outage

Описание: On October 27, 2026, GitHub executed what was intended to be a routine online table migration across two MySQL clusters. Seven tables were moving live using their open-source tool, gh-ost. But for over an hour, roughly one in 40 authenticated Git operations returned an HTTP 404—not a server error or timeout, but a confident claim that valid repositories and installations did not exist.

The cause was not a bad deploy, broken code, or failed disk. It was a scheduled cleanup job that deleted approximately 59 million rows from the installation records table while the migration was actively copying data. As 59 million row-based binlog delete events flooded the replication stream between IAD and Azure Central US, replication lag surged to one hour. When automated read cutovers progressed according to the calendar rather than true replication health, GitHub cut over 100% of read traffic to a database operating an hour in the past.

In this episode of Skildcore, we dissect the minute-by-minute timeline, explain why gh-ost's triggerless binlog architecture fell behind, detail why missing installation rows masqueraded as 404s, and break down the two-line process fix GitHub deployed.

---
TIMESTAMPS:
0:00 - The Routine Maintenance Trap: 1 in 40 Git Operations Return 404
0:50 - The Collision: 59 Million Deletes During an Online Migration
1:19 - Chapter 01: The Postmortem Numbers (gitstatus.com Incident Window)
2:21 - Chapter 02: Under the Hood of gh-ost (Triggerless Migrations & Binlogs)
3:48 - Chapter 03: The Collision (59 Million Row Deletion Storm)
4:46 - The 1-Hour Lag: Inter-Cluster Replication Between IAD and Azure
5:17 - Chapter 04: The Cutover Timeline (04:44 to 08:21 UTC Breakdown)
6:06 - Why Did It Cut Over? The Schedule-Driven Automation Failure
6:27 - Chapter 05: The Mechanism: Why Missing Rows Answered as 404 Not Found
7:48 - Scrupulous SRE: What GitHub Confirmed vs What Is Inferred
9:09 - The Fix: Two Boring Lines of Defense
9:38 - Architectural Verdict: Never Let Cutover Be a Time, Make It a Predicate

---
KEY TECHNICAL TAKEAWAYS:
• Triggerless migration mechanics: Unlike pt-online-schema-change, gh-ost operates without database triggers, tailing MySQL row-based binary logs as an asynchronous replica to eliminate write-path overhead.
• The delete storm amplification: A batch cleanup deleting 59 million rows emits 59 million discrete row events into the binary log, saturating the inter-cluster replication link between IAD and Azure Central US with an hour of queue lag.
• Absence looks like 404: Because the new cluster was serving an hour-stale snapshot, server-to-server app tokens created during the window had no corresponding installation record on the replica, faithfully returning 404 Not Found.
• Blast radius containment: The failure strictly impacted tokens querying the migrated installation records table; GitHub Actions tokens and unaffected token types were completely isolated.
• Predicate-based cutovers: Migration cutover must never be a clock time or calendar event; cutover must be a runtime predicate gated strictly on replication lag dropping below a verified threshold.
• Pre-flight dependency scanning: Online migrations must hold a lock or dependency declaration against the table to automatically park scheduled batch jobs and crons before the first row copies.

CITATIONS & POSTMORTEM SOURCES:
• GitHub Availability Report: Authenticated Git operations returned 404 (gitstatus.com, Oct 27, 2026)
• Open Source Systems: GitHub gh-ost (Triggerless Online Schema Migration for MySQL)
• Infrastructure: MySQL clusters across IAD and Azure Central US

#GitHub #MySQL #DatabaseReliability #SRE #DevOps #SystemArchitecture #Postmortem #gh_ost #DistributedSystems #Skildcore