GitHub described June’s infrastructure progress before its incident write-ups: monolith traffic in Azure peaked at 45% in Central US, lower than hoped because GitHub paused the ramp for roughly a month after a stability incident on May 21 made clear the environment wasn’t ready for more traffic, restarting the ramp on June 17 with a new per-turnup stability gate. Git in Azure grew from 30% to a peak of 43% over the month, missing GitHub’s June target of 50%. New extracted services, pullsd and reposd, began handling production traffic, and two-person confirmation is now required end-to-end for interactive production access and ChatOps changes.
In June, GitHub experienced six incidents that resulted in degraded performance across GitHub services.
Incident 1: June 04, 17:30 UTC (1 hour 25 minutes)
From 17:30 to 18:55 UTC, Copilot code review experienced elevated failures for review requests on github.com; affected users saw “Copilot ran into an error” on pull requests when requesting a code review. An average of 81.6% of requests failed, with a peak failure rate of 93.9%, and approximately 36,800 code review requests failed in total.
Root Cause: A newly released dependency used by the Copilot code review processing workflow introduced an incompatibility with the runtime environment. Because the workflow automatically consumed the latest release, the incompatible version was picked up without sufficient compatibility validation, and affected review jobs did not fail fast, with many continuing to run until they timed out.
Remediation: GitHub mitigated the incident by removing the problematic dependency version and redeploying the affected processing service; new code reviews began recovering at 18:44 UTC and the failure rate returned to baseline by 18:55 UTC. GitHub is pinning the dependency version instead of automatically consuming the latest release, adding compatibility checks, improving fast-failure behavior, adding shorter timeout controls, and improving monitoring for review completion failures.
Incident 2: June 08, 06:30 UTC (2 hours 6 minutes)
Between approximately 06:30 and 08:36 UTC, signed-out users experienced sustained elevated HTTP 504 errors accessing pull requests, issues, releases, patch diffs, and other related github.com pages. Approximately 17% of unauthenticated requests to affected endpoints returned gateway timeout errors, peaking at roughly 34% around 06:50 UTC. Signed-in users were not affected.
Root Cause: A significant increase in abusive, automated anonymous traffic to specific github.com endpoints degraded the dedicated pool of web application servers that serve unauthenticated requests, causing requests to queue beyond timeout thresholds.
Remediation: GitHub identified the anomalous traffic pattern and applied targeted blocks at the load balancer and application layers; affected services were fully restored by 08:36 UTC. GitHub is improving automated detection and blocking for these traffic patterns, improving its emergency traffic-blocking deployment path, and evaluating routing changes for endpoints used by both signed-out users and automated workflows.
Incident 3: June 10, 15:05 UTC (1 hour 20 minutes)
Between 15:05 and 16:25 UTC, GitHub API services experienced degraded availability due to sporadic authentication failures affecting approximately 9% of requests across both REST and GraphQL. Customers experienced intermittent “logged out” behavior as erroneous 401 responses caused app integrations to trigger repeated authentication flows, and affected requests experienced approximately 800ms of additional latency.
Root Cause: A memcached proxy service rollout to GitHub’s internal API infrastructure caused the authentication service to pick up an incorrect host configuration, leading to intermittent authentication lookup failures.
Remediation: GitHub mitigated the incident by deploying a configuration change to the memcached service to use the correct host. GitHub plans to migrate its authentication system to the new caching infrastructure and is improving how the gateway distinguishes transient authentication-system errors from genuinely invalid credentials.
Incident 4: June 16, 17:20 UTC (55 minutes)
Between 17:20 and 18:15 UTC, the Opus 4.8 model experienced degraded availability in GitHub Copilot; some requests failed or errored while other Copilot models remained available as alternatives.
Root Cause: An issue with an upstream model provider.
Remediation: GitHub enabled degraded-mode messaging while the issue was ongoing; the upstream provider resolved the issue and GitHub monitored until success rates returned to normal. GitHub is reducing reliance on any single inference provider for a given model and balancing capacity across providers so traffic can fail over during an upstream outage.
Incident 5: June 17, 03:50 UTC (54 minutes)
Between approximately 03:50 and 04:44 UTC, GitHub Copilot was degraded and most frontier chat models were temporarily unavailable across all regions; affected models either disappeared from the model picker or returned a “model not available” error. The incident occurred during off-peak hours, limiting the number of customers affected.
Root Cause: A configuration change that GitHub’s production system deemed invalid.
Remediation: GitHub mitigated the incident by reverting the configuration change, after which affected models returned automatically as the service reloaded the previous configuration. GitHub is working to roll out configuration changes gradually with stronger validations, alert on sudden drops in the number of available models, and automatically roll back configuration changes that trigger these alerts.
Incident 6: June 25, 17:33 UTC (23 minutes)
Between 17:33 and 17:55 UTC, GitHub’s background job service experienced degradation that increased delays to pull requests, repository pushes, actions workflows, and webhooks, with delays peaking at 7 minutes.
Root Cause: Underlying hypervisor issues combined with an incoming traffic spike caused service timeouts, leading to a connection storm and continual rebalances.
Remediation: GitHub mitigated the issue by replacing the impacted node at 17:49 UTC, after which all services recovered by 18:07 UTC. GitHub has made background job processing more resilient to sudden traffic spikes, reduced the possibility of connection churn in degraded cases, removed the co-location of critical nodes so one unhealthy node cannot affect others, and added earlier alerting on the conditions that led to this degradation.