In April, GitHub experienced three incidents causing degraded service performance.
April 11 Incident (03:05 UTC, lasting 39 minutes)
Between 03:05 and 03:44 UTC, Codespaces create and start operations failed for approximately 75% of users.
Root Cause: Manual configuration changes to an internal dependency that escaped GitHub’s test coverage.
Resolution: Monitors triggered, enabling rapid triage and reversion of the changes to restore service health.
Preventive Measures: GitHub is building additional gates, safer mechanisms for testing, and rolling out such configuration changes.
April 23 Incident, first wave (07:00 UTC, lasting 20 minutes)
Between 07:00 and 07:20 UTC, multiple GitHub services experienced error rates of 2-5%, causing intermittent service disruption.
Root Cause: Resource contention on database hosts, triggered by an interaction between query load and an ongoing schema change that led to connection saturation.
Resolution: Completion of the schema migration restored normal operations.
Preventive Measures: GitHub is reviewing database capacity, improving monitoring/alerting, and implementing safeguards to reduce detection time.
April 23 Incident, second wave (19:13 UTC, lasting approximately 2 hours 58 minutes)
Between 19:13 and 22:11 UTC, the GitHub Migration service failed, affecting 837 migrations across 57 organizations.
Root Cause: A configuration change removed repository migration worker access.
Resolution: Restoring access to the affected workers resumed normal operations.
Preventive Measures: GitHub is implementing enhanced test coverage and refining monitoring thresholds.