In September, GitHub experienced three incidents affecting service availability.
September 15 Incident (17:55 UTC, lasting 25 minutes)
Between 17:55 and 18:20 UTC, Copilot experienced degraded availability for the majority of its features.
Root Cause: A partial deployment of a feature flag to a global rate limiter. The flag triggered behavior that unintentionally limited 100% of requests, returning 403 errors. The flag was meant to scale down rate limiting for a subset of users, but unintentionally put the rate limiting configuration into an invalid state.
Resolution: Reverting the feature flag resulted in immediate recovery.
Preventive Measures: GitHub is adding traffic anomaly monitors for early issue detection, and increasing coverage of rate limit scaling tests to strengthen pre-production validation.
September 23-24 Incident (15:29 UTC September 23 through 15:12 UTC September 24, approximately 130 minutes total, with peak customer delays of approximately 50 minutes)
Email deliveries were delayed, resulting in significant delays for most types of email notifications.
Root Cause: An unusually high volume of traffic, which caused resource contention on some of GitHub’s outbound email servers.
Resolution: Configuration updates to better allocate capacity when there is a high volume of traffic.
Preventive Measures: GitHub is updating its monitors to improve detection capabilities.
September 29 Incident (16:26 UTC, lasting 67 minutes)
Between 16:26 and 17:33 UTC, the Copilot API experienced partial degradation, causing intermittent erroneous 404 responses for an average of 0.2% of GitHub MCP server requests, peaking around 2%.
Root Cause: An upgrade of an internal dependency, which exposed a misconfiguration in the service.
Resolution: GitHub rolled back the upgrade to address the misconfiguration.
Preventive Measures: GitHub fixed the configuration issue and will improve documentation and rollout process.