During January, GitHub experienced one significant incident that degraded the GitHub Actions service.
January 28 Incident (04:21 UTC, lasting 3 hours 53 minutes)
Company monitoring systems detected abnormal error rates affecting Actions, causing “failure or delay of some queued jobs for a period of time.” Jobs queued during the outage completed successfully once the problem was resolved.
Root Cause: “We identified the issue as caused by an infrastructure error in our SQL database layer.” This database problem impacted a core microservice responsible for authentication and communication between Actions microservices, disrupting job queuing across the service.
Why Detection Failed: Normally, automated systems detect unhealthy databases and initiate failover with minimal impact. In this case, “the failure pattern was not recognized by the automated processes, and telemetry did not show issues with the database,” extending the time needed to identify and resolve the issue.
Preventive Measures: “We are updating the automation processes in our SQL database layer to improve error detection and failovers. Furthermore, we are continuing to invest in localizing failures to minimize the scope of impact.”