In March, GitHub experienced six incidents that resulted in degraded performance across GitHub services. The report also provides follow-up root-cause detail on a February incident affecting GitHub Codespaces.
Follow-up: February 28 Incident (15:42 UTC, lasting 1 hour and 26 minutes)
Higher-than-normal failure rates for Codespaces creation/startup occurred in the East US region.
Root Cause: Slower than normal VM allocation time from the cloud provider.
Resolution: GitHub redirected codespace creations to a secondary region as mitigation.
Preventive Measures: Tuned monitoring for faster detection and architectural changes enabling automatic failover.
March 1 Incident (12:01 UTC, lasting 3 hours and 6 minutes)
Elevated latency for requests to Container, NPM, NuGet, and RubyGems Packages registries affected 0.5% of requests initially, peaking at 10%.
Root Cause: An unhealthy disk on a database VM node caused significant performance degradation at the OS level, creating connection delays and slow query execution.
Resolution: The team performed a database failover.
Preventive Measures: Planned MySQL infrastructure migration with improved observability tools.
March 2 Incident (23:37 UTC, lasting 2 hours and 18 minutes)
Failed requests for GitHub Actions workflows resulted from TLS verification failures.
Root Cause: An unexpected SSL certificate bound to GitHub’s CDN IP address, caused by the CDN provider’s configuration change.
Resolution: The team removed the problematic certificate binding.
Preventive Measures: Evaluating multiple DNS/CDN providers.
March 15 Incident (14:07 UTC, lasting 1 hour and 20 minutes)
Increased latency occurred for requests to Container, NPM, NuGet, and RubyGems package registries during routine maintenance.
Root Cause: A slow running query blocked all database resources.
Resolution: The team killed the slow query and restarted the database.
Preventive Measures: Paused maintenance pending investigation and enhanced safety checks.
March 27 Incident (12:25 UTC, lasting 1 hour and 33 minutes)
Impact affected Pages, Codespaces, and Issues. Due to the recency of the incident, GitHub said it was still investigating the contributing factors.
March 29 Incident (14:21 UTC, lasting 4 hours and 29 minutes)
Pages, Codespaces, and Actions experienced impact. Investigation was ongoing at the time of the report.
March 31 Incident (01:16 UTC, lasting 52 minutes)
Git operations, Issues, and pull requests were affected. Detailed analysis was pending at the time of the report.