The April report documents four incidents in April 2023 and provides follow-up root-cause detail on three incidents from March 2023 that caused degraded performance across GitHub services.
Follow-up: March 27 Incident (12:25 UTC, lasting 1 hour and 33 minutes)
Starting at 12:14 UTC, users experienced degraded performance with Git Operations, GitHub Issues, pull requests, GitHub Actions, API requests, Codespaces, and Pages. Services were publicly statused as yellow then red, with full restoration at 13:17 UTC.
Root Cause: A change in a frequently-used database query increased lock contention and resource exhaustion during load spikes, causing database crashes and failover complications.
Resolution: The problematic query was disabled via feature flag and refactored.
Preventive Measures: Added monitoring of database resources, enhanced query evaluation procedures, and improved documentation.
Follow-up: March 29 Incident (14:21 UTC, lasting 4 hours and 57 minutes)
At 14:10 UTC, GitHub Actions workflows stopped progressing. Recovery began at 14:57 UTC but degraded again before fully recovering at 19:03 UTC.
Root Cause: A degraded database cluster caused by a new load source from a background job querying that database cluster, which maxed out database transaction pools, combined with underprovisioning of vtgate proxy instances.
Resolution: Job processing was throttled and capacity was added.
Preventive Measures: Improved telemetry and alerting to identify blocking queries faster, and closer monitoring of cluster health.
Follow-up: March 31 Incident (01:07 UTC, lasting 2 hours)
Beginning at 00:06 UTC, a small percentage of users received 500 errors on pull request file pages. Engineering identified the cause at 01:07 UTC; the fix deployed by 02:07 UTC.
Root Cause: An expiry date in the configuration of a notification was set incorrectly, causing a constant that was still referenced in code to no longer be available.
Preventive Measures: GitHub audited existing notice expiry dates, prevented invalid configurations, and improved test coverage.
April 18 Incident (09:28 UTC, lasting 11 minutes)
At 09:22 UTC, users accessing issues or pull request entities received 5xx responses. The issue self-healed at 09:33 UTC; services were statused green at 09:42 UTC.
Root Cause: A planned database infrastructure change involved deleting nodes receiving live traffic, creating an 11-minute window where database cluster requests failed.
Preventive Measures: GitHub reviewed and improved change management processes, updated monitoring guidelines to check traffic patterns before disruptive actions, added review steps, and implemented a new checklist for infrequent administrative changes.
April 26 Incident (23:26 UTC, lasting 1 hour and 4 minutes)
GitHub Copilot experienced an outage from 23:26 UTC until resolution at 00:29 UTC. Investigation was ongoing at the time of publication.
April 27 Incident (08:59 UTC, lasting 57 minutes)
A GitHub Packages outage occurred at 08:59 UTC, resolved at 09:56 UTC. Investigation was ongoing at the time of publication.
April 28 Incident (12:26 UTC, lasting 19 minutes)
GitHub Codespaces had degraded availability beginning at 12:26 UTC, resolved at 12:45 UTC. Investigation was ongoing at the time of publication.