In September, GitHub experienced three incidents that resulted in degraded performance across services.
Incident 1: September 16, 2024 (21:11–22:08 UTC, lasting 57 minutes)
Affected Services: GitHub Actions and GitHub Pages.
Root Cause: A misconfiguration in the service that manages runner connections led to CPU throttling and performance degradation.
Impact: Actions jobs experienced average delays of 23 minutes, with some reaching 45 minutes. At peak, 80% of runs faced delays exceeding five minutes.
Resolution: Runner connections were diverted away from misconfigured nodes starting at 21:16 UTC.
Preventive Measures: Monitoring improvements were implemented to reduce detection and mitigation time.
Incident 2: September 24, 2024 (08:20–09:04 UTC, lasting 44 minutes)
Affected Services: GitHub Codespaces.
Root Cause: A network connectivity interruption caused by Source Network Address Translation (SNAT) port exhaustion following a deployment, resulting in approximately a 25% error rate.
Resolution: Port allocations were increased to provide buffer for increased outbound connections.
Preventive Measures: Scaling of outbound connectivity and improved network capacity monitoring were planned.
Incident 3: September 30, 2024 (10:43–11:26 UTC, lasting 43 minutes)
Affected Services: GitHub Codespaces (Central India region only).
Root Cause: Storage capacity constraints in the region; a bug prevented available capacity from being utilized.
Resolution: Create requests were temporarily redirected to other regions; additional storage was added; the capacity utilization bug was fixed.