In February, GitHub experienced three incidents that resulted in degraded performance across GitHub services. The report also provides follow-up root-cause detail on two incidents from January.
Follow-up: January 30 Incident (21:31 UTC, lasting 35 minutes)
Container registry experienced increased 500 errors, causing most GitHub Pages builds and GitHub Packages requests to fail.
Root Cause: A configuration change enforced authentication on Redis connections at 21:30 UTC. The deployment file had a hardcoded connection string that prevented client authentication.
Resolution: The Redis authentication configuration was reverted at 22:12 UTC. Container registry recovered within two minutes, and GitHub Pages returned to healthy status by 22:21 UTC.
Preventive Measures: Improved secrets management in Container registry Redis deployment configurations and added test coverage for authenticated Redis connections.
Follow-up: January 30 Incident (18:35 UTC, lasting 7 hours)
A deployment modified compression settings on source code downloads, altering archive file checksums while keeping contents unchanged.
Root Cause: Communities had built dependencies on the exact byte layout of archived files, creating unforeseen consequences.
Resolution: The change was reverted and affected communities were notified.
Preventive Measures: Adding checksum validation to the workflow and implementing improvements in Git deployment procedures.
February 7 Incident (21:30 UTC, lasting 20 hours and 35 minutes)
Failures occurred creating, starting, and connecting to GitHub Codespaces in Southeast Asia due to a cloud provider datacenter outage.
Root Cause: A regional datacenter outage from the cloud infrastructure provider.
Resolution: Codespace creations were redirected to secondary locations. Existing codespaces recovered automatically when the datacenter came back online.
Preventive Measures: Evaluating expanded regional redundancy and making architectural changes to better handle temporary regional outages.
February 18 Incident (02:36 UTC, lasting 2 hours and 26 minutes)
System-wide 500 errors occurred, and users experienced sign-in failures to GitHub.com.
Root Cause: Unhealthy deployments of sharding middleware caused MySQL connectivity issues. A re-deployment at 03:30 UTC propagated the problem to all Kubernetes pods, corrupting database topology in 30% of deployments.
Resolution: The correct database topology was applied to all deployments at 04:30 UTC, resolving errors by 05:00 UTC and restoring user sign-in capability.
Preventive Measures: Added monitoring to detect database topology errors early, improved observability around topology reloads, and committed to a deeper architectural review.
February 28 Incident (16:05 UTC, lasting 1 hour and 26 minutes)
Degraded performance for GitHub Codespaces was reported. GitHub said it was “still investigating the contributing factors” due to the recency of the incident and promised more detailed analysis in the next monthly report.
Resolution: The incident was resolved at 17:31 UTC.