In March, GitHub experienced recurring database stability issues along with a separate incident during a data migration.
Database Stability Issues (multiple incidents throughout March, including March 14, March 24, and March 28)
GitHub experienced recurring incidents impacting core services including Git, issues, and pull requests. The mysql1 database cluster experienced resource contention during peak traffic periods. Write operations were significantly affected, though some read operations remained functional. GitHub Actions queues became saturated after other services recovered, causing multi-hour outages.
Root Cause: The underlying issue was resource contention in the mysql1 cluster during peak load bursts. The database had insufficient headroom and was susceptible to poorly performing queries. Query performance issues combined with low capacity margins created vulnerability to outages.
Remediation:
- Implemented query optimizations that reduced queries per second by over 50% and transaction volume by 70% at peak load (March 14-28)
- Proactively throttled webhooks and Actions services on March 14 at 14:43 UTC to prevent outages
- Paused migrations and team synchronization during peak hours
- Scheduled maintenance windows for three services starting March 24
- Reduced database performance alerting thresholds
- Accelerated work to shard the mysql1 cluster further
- Dedicated an engineering team to review internal processes and observability
Actions and Checks Data Migration Failure (March 29, 10:26 UTC, lasting 57 minutes)
During migration of GitHub Actions and checks data to a dedicated, sharded database cluster, a misconfiguration caused application errors. Some queued jobs failed or were delayed. After reverting changes, jobs queued during the incident ran successfully post-resolution.
Root Cause: Insert queries failed on the new database cluster due to missing permissions, though update and delete queries processed correctly.
Remediation:
- Reverted changes immediately upon detecting Actions SLO breaches
- Paused further migration attempts pending full investigation
- Scheduled future migrations outside peak traffic hours (avoiding the 12:00-21:00 UTC window)
- Committed to scrutinizing operational workflows for database operations
- Planned a configuration and topology audit across the environment
- Extended the pre-migration checklist to include thorough permissions verification