In January, GitHub experienced three incidents that caused degraded performance across services.
Incident 1: January 9, 2024 (12:20–14:40 UTC, lasting 140 minutes)
Root Cause: A host upgrade reduced capacity temporarily. The configured connection limit was lower than it should have been, causing bottlenecks.
Impact: One site experienced elevated latency. An average of 5% and peak of 10% of requests failed or timed out, affecting the Git backend and other services.
Preventive Measures:
- Increased connection limits
- Improved monitoring of connection limit behavior
- Modified upgrade procedures to reduce capacity reduction risk
Incident 2: January 21, 2024 (02:01–09:34 UTC, lasting 7 hours 3 minutes)
Root Cause: Operational issues with compute and storage resources affected Codespaces functionality.
Impact: Approximately 25% of customers were impacted, primarily in the East US and West Europe regions. Users could not create or resume Codespaces.
Preventive Measures:
- Rerouted traffic to less affected regions
- Improved alerting and regional outage resiliency
Incident 3: January 31, 2024 (starting 12:30 UTC, lasting 147 minutes)
Root Cause: A load balancer infrastructure change for IPv6 preparation converted IPv4 addresses to IPv4-mapped formats, causing IP Allow List functionality to reject valid addresses as unauthorized.
Impact: Request error rate peaked at 0.23% of all requests.
Preventive Measures:
- Deployed fixes to remediate the issue
- Enhanced testing and monitoring procedures