In September, GitHub experienced two incidents that resulted in degraded performance across services.
Incident 1: September 5, 2023 (16:24–16:43 UTC, lasting 19 minutes)
Root Cause: A primary database host experienced a file system write error affecting a shared datastore. The disk storage for the operating system became unreachable, causing a partial host failure that the automatic failover system could not detect.
Impact: SAML login, GitHub Actions, GitHub Issues, pull requests, GitHub Pages, GitHub API, Webhooks, GitHub Codespaces, and GitHub Packages were affected.
Resolution: Manual failover to a healthy host was initiated 17 minutes after the initial alert and completed 2 minutes later.
Preventive Measures: GitHub improved automation to detect and initiate failover for partial host failures and patched a fix addressing resource contention.
Incident 2: September 19, 2023 (20:36 UTC, degradation lasting 3.5 hours; full data recovery completed at 04:28 UTC, 7 hours 30 minutes)
Root Cause: During a primary datastore migration for GitHub Projects, a misconfigured index constraint rendered the Projects database fully unavailable. Database replication interruption from remediation steps additionally impacted Git Operations, APIs, and GitHub Issues.
Resolution: GitHub restored Projects data to its pre-incident state by 00:06 UTC. Recovered user-created project data became available by 04:28 UTC.
Preventive Measures: GitHub is implementing enhanced migration validation and constraint evaluation.