The May report documents seven incidents across April and May 2023: four May incidents and follow-up root-cause detail on three April incidents that caused degraded performance across GitHub services.
Follow-up: April 26 Incident (23:11 UTC, lasting 51 minutes)
GitHub Copilot code completions degraded for a subset of users across all three regions: US Central, US East, and Switzerland North. Peak impact was 6% of completion requests failing.
Root Cause: A faulty configuration change by an automated maintenance process placed defective nodes into service before rollout halted.
Resolution: Automated traffic rerouting mitigated the impact.
Preventive Measures: Reduce batch size and iteration speed of maintenance processes; adjust alerting thresholds for faster detection.
Follow-up: April 27 Incident (08:59 UTC, lasting 57 minutes)
GitHub Packages experienced slow or failed upload and download requests due to connection errors on the primary database node.
Root Cause: A database connection spike caused availability degradation.
Resolution: A manual database restart at 09:56 UTC restored service. The database was migrated to a more robust platform on May 2, 2023.
Follow-up: April 28 Incident (12:26 UTC, lasting 19 minutes)
GitHub Codespaces users in the East US region experienced failures creating and resuming codespaces.
Root Cause: The cloud provider experienced an outage in the East US region; virtual machines had internal operation errors.
Resolution: Regional failover redirected East US operations to East US 2 at 12:45 UTC. When the provider outage resolved, seamless redirection occurred with no downtime.
May 4 Incident (15:53 UTC, lasting 30 minutes)
Degraded performance occurred across Git Operations, GitHub APIs, Issues, Pull Requests, Webhooks, Actions, Pages, Codespaces, and Copilot.
Root Cause: Connection pool exhaustion on the proxy layer; prior configuration updates had been applied inconsistently.
Resolution: A primary failover was performed on the repositories database cluster.
Preventive Measures: Audited and fixed proxy configurations; updated automation for dynamic configuration changes without disruption.
May 9 Incident (11:27 UTC, lasting 10 hours and 44 minutes)
Git data read/write failures occurred from 11:27–12:33 UTC affecting multiple services. Repositories and Pull Requests required additional recovery time until 21:20 UTC.
Root Cause: A database bug in the current version was more likely to trigger under GitHub’s custom cluster configuration.
Resolution: Updated configuration to match other clusters.
Preventive Measures: The bug was reported to maintainers, accepted as a private issue, with a fix expected in the July release. Removed special-case database configurations; expanded graceful degradation tooling.
May 10 Incident (12:38 UTC, lasting 11 hours and 56 minutes)
GitHub Apps auth token issuance failed, impacting Actions, APIs, Codespaces, Git Operations, Pages, and Pull Requests.
Root Cause: A significant write latency spike on the shared permissions database cluster, caused by expensive database write transactions from new API call data patterns.
Resolution: Identified and blocked the problematic data shapes.
Preventive Measures: Conducted an endpoint audit for similar patterns; improved observability of API errors; improved MySQL write pattern diagnosis tools; enhanced playbooks and incident response training.
May 16 Incident (21:07 UTC, lasting 25 minutes)
Issues, Pull Requests, and Git Operations were unavailable; APIs, Actions, Pages, and Codespaces were partially unavailable.
Root Cause: A primary key-value database cluster hardware crash left the cluster unable to automatically failover due to data loss risk.
Resolution: Manual failover was triggered 11 minutes after the crash.
Preventive Measures: Improving alerting; reducing dependency on the cluster as a single point of failure.