In April, GitHub experienced four incidents that resulted in degraded performance across services.
Incident 1: April 5 (08:11–08:58 UTC, lasting 47 minutes)
Root Cause: A database load balancer modification triggered connection failures affecting multiple critical databases in one data center.
Impact: Web request errors peaked at 6%, API errors at 10%, and over 100,000 GitHub Actions workflows failed to initiate.
Resolution: The problematic change was rolled back. GitHub implemented new measures to detect similar problems earlier in the deployment pipeline.
Incident 2: April 10 (08:18–09:38 UTC, lasting 120 minutes)
Root Cause: An unbounded query overloaded the primary database instance.
Impact: 17% failure rate for web-based file editing, 1.5–8% failure rates for repository operations, severe impacts on issue/pull request authoring, and 5% GitHub search failures.
Resolution: The instance was scaled up and an improved query version deployed to read replicas. GitHub is working to reduce reliance on the affected primary database.
Incident 3: April 10 (18:33–19:03 UTC, lasting 30 minutes)
Root Cause: A compute-intensive database query prevented a critical cluster from serving other requests.
Impact: GitHub Actions delays/failures, API timeouts, all GitHub Pages deployments failed, Git HTTP errors, Issues latency increases, and Codespaces timeouts.
Resolution: The offending query was rolled back. Detection gaps in CI testing were identified and addressed; services were improved for resilience and regression detection.
Incident 4: April 11–14 (lasting 3 days, 4 hours, 23 minutes)
Root Cause: Increased shared resource pool usage combined with an unhealthy internal job queue prevented email processing.
Impact: Email delivery delays of up to two hours, particularly affecting password resets and device verification; users without 2FA could not complete account recovery.
Resolution: Queue-bypass capability for time-sensitive emails was implemented; detection mechanisms for anomalous delivery were improved; the unhealthy queue was paused.