GitHub reported on its ongoing infrastructure investments before covering May’s incidents: the company is now serving 40% of monolith traffic from Azure (up from 8% in February), with Git traffic at 30% and repository replication at 99%, more than doubling effective capacity in four months. GitHub is also completing isolation of its primary database cluster—splitting users, authentication, and authorization into independent domains so a problem in one can no longer cascade across the platform—and rolling out stateless authentication tokens to eliminate per-request database lookups that had amplified pressure during traffic spikes.
In May, GitHub experienced nine incidents that resulted in degraded performance across GitHub services.
Incident 1: May 04, 15:45 UTC (55 minutes)
Between 15:34 and 16:40 UTC, github.com experienced a service disruption producing elevated latency and increased request failures across a broad set of customer-facing services, with total customer impact lasting approximately one hour and six minutes. Pull requests was the most significantly impacted service, statused Red for the duration of peak impact; issues, actions, webhooks, and Git operations experienced elevated latency and intermittent errors. At peak, approximately 1.3% of requests returned a 5xx response, averaging around 0.46% across the incident.
Root Cause: A routine online schema migration running against a large, heavily-accessed database table had been progressing without issue for several hours, but as traffic ramped up toward the weekly peak, the combined load from the migration and normal production traffic saturated database connection capacity, producing query contention on a primary database and cascading timeouts.
Remediation: GitHub detected the incident within approximately three minutes and paused the contributing migration, after which dependent services recovered; time to mitigation was approximately 33 minutes, with full resolution about 30 minutes later. GitHub is aligning migrations against large, high-traffic tables with low-traffic windows, using dynamic throttling that adapts to live cluster load, adding automated circuit breakers to pause in-flight migrations when latency or connection utilization crosses safe thresholds, and reviewing connection-pool capacity.
Incident 2: May 05, 13:37 UTC (3 hours 49 minutes) and Incident 3: May 06, 07:19 UTC (2 hours 25 minutes)
On May 5, from 13:22 to 17:05 UTC, GitHub Actions hosted runners in the East US region were degraded: approximately 13.5% of jobs requesting a standard runner failed, and ~16% of requested larger runners with private networking pinned to East US failed or were delayed by more than five minutes. Copilot code review requests were also impacted, with approximately 8,500 code review requests timing out. On May 6, from 06:45 to 09:15 UTC, GitHub Actions Standard Ubuntu hosted runners were again degraded, with approximately 17.1% of jobs requesting a standard runner failing. The two events were connected: remediation work performed after the May 5 incident introduced the configuration issue that triggered the May 6 incident.
Root Cause: The May 5 incident was triggered by a routine scale-up operation for hosted runner VMs in East US that hit an internal rate limit when VM creates pulled images from storage; existing backoff logic was not triggered because of the response code returned. The May 6 incident was caused by unexpected configuration data introduced during remediation work for the previous day’s incident, which blocked new allocations as daily load ramped up.
Remediation: GitHub mitigated the May 5 incident by reducing load to allow recovery and processing queued work; by 15:34 UTC queued/failed assignments were mostly mitigated, with full recovery at 17:05 UTC. GitHub removed the problematic configuration data at 08:51 UTC on May 6, allowing allocations to resume. GitHub is improving throttling behavior when limits occur, improving controls to more quickly mitigate similar situations, reviewing all limits end-to-end for similar operations, updating filter logic for allocation data to be resilient to abnormal data shapes, and improving monitoring to alert when allocations are blocked.
Incident 4: May 06, 11:21 UTC (38 minutes)
Between 11:02 and 11:13 UTC, users were unable to start or view Copilot cloud agent or remote sessions; all requests to the session API returned errors.
Root Cause: A configuration change to the service’s network routing inadvertently removed the ingress path for the service.
Remediation: GitHub reverted the change at 11:13 UTC, restoring service; the incident remained open until 11:59 UTC while the team verified full recovery. GitHub is improving its deployment validation process to prevent similar configuration changes from impacting production traffic.
Incident 5: May 06, 15:25 UTC (3 hours 39 minutes)
Between 15:12 and 19:02 UTC, creation of new pull request review threads on github.com failed, including new line comments and file comments. Existing pull requests and previously created comments were unaffected.
Root Cause: A 32-bit integer key reached its maximum value in a Vitess lookup table used during pull request thread creation. The primary table had been migrated to a 64-bit integer key, but the Vitess lookup table remained 32-bit; once values passed the available 32-bit ID space, new review thread creation began failing at a near-100% rate.
Remediation: GitHub mitigated the issue by updating the impacted lookup table definitions across all shards to use 64-bit integer column types, restoring normal operation once schema changes completed globally. GitHub is expanding existing monitoring of database columns to include Vitess lookup tables to enable earlier detection of tables approaching a column size limit.
Incident 6: May 07, 05:02 UTC (1 hour 54 minutes)
Between 04:12 and 06:13 UTC, Copilot cloud agent and Copilot code review agent sessions for pull requests were delayed or failed to start; new coding agent sessions triggered by pull requests were not created, while review agent sessions dropped by ~50% from baseline.
Root Cause: Follow-up recovery work from the May 6 pull request incident involved a large database migration that caused replication delays on several replica hosts. Although those replicas were not serving user traffic, GitHub’s safeguards correctly treated elevated replication lag as a signal to slow down writes to the affected database cluster, temporarily delaying pull request background processing that sends the internal events Copilot agents use to begin work.
Remediation: The system recovered once replication lag returned to normal and pull request processing resumed. GitHub is making critical background processing more resilient to replication lag so it degrades gracefully and keeps essential events flowing under strain.
Incident 7: May 15, 08:13 UTC (35 minutes)
From 07:43 to 08:48 UTC, GitHub Actions experienced a degradation causing workflow runs to fail or experience delayed starts for a subset of customers; at peak, 42% of Actions runs failed. Downstream services including GitHub Pages and Copilot cloud services were also impacted.
Root Cause: The incident was triggered by a planned failover of supporting infrastructure used by GitHub Actions; during that operation, an automated service discovery update did not propagate correctly, causing traffic to be routed incorrectly and increasing request timeouts in a core dependency for workflow orchestration.
Remediation: At 08:12 UTC, responders manually corrected the service discovery routing issue; the incident was marked resolved at 08:48 UTC. GitHub is implementing failover guardrails that validate service discovery state before completing failover operations, strengthening verification checks, and improving dependency resilience to reduce timeout cascades during infrastructure events.
Incident 8: May 26, 10:57 UTC (2 hours 21 minutes)
Between 10:40 and 12:56 UTC, GitHub Actions jobs were degraded: from 10:40 to 12:16 UTC all newly queued runs failed to start, and from 12:16 to 12:56 UTC runs requiring downloading actions continued to fail. GitHub Pages, Copilot code review, Copilot coding agent, Octoshift, and GitHub Enterprise Importer were also impacted. A small number of issues, pull requests, comments, and discussions were marked as hidden when the service account was disabled, but no data was lost and all content was restored.
Root Cause: GitHub’s automated account review system incorrectly suspended the service account used by GitHub Actions to authenticate workflow runs and download actions.
Remediation: GitHub restored the account at 12:16 UTC, marked it exempt from further automated review at 12:20 UTC, and redeployed a related service at 12:48 UTC to flush cached account state; full recovery was confirmed at 12:56 UTC. GitHub added an allowlist of service accounts that cannot be suspended by automated systems, and is improving diagnostic tooling for accounts and reducing cache propagation delays.
Incident 9: May 28, 19:01 UTC (1 hour 40 minutes)
Between 18:27 and 20:41 UTC, the GitHub Copilot service was degraded due to an issue with the Responses API of an upstream provider affecting the GPT-5.2, GPT-5.3-Codex, GPT-5.4, and GPT-5.5 models; requests routed to these models returned elevated error rates, also affecting Copilot coding agent and Copilot code review. No other models were impacted.
Root Cause: An issue with the Responses API of an upstream model provider.
Remediation: GitHub mitigated the incident by shifting traffic away from the affected models while the upstream provider deployed a fix. GitHub is working to improve automated failover for affected models and strengthen monitoring.