In April, GitHub experienced 10 incidents that resulted in degraded performance across GitHub services. To increase transparency, at the end of April GitHub released a separate blog post covering the major incidents on April 23 and April 27, and took steps to bring more detail to the GitHub status page.
Incident 1: April 01, 15:02 UTC (8 hours 43 minutes)
Between 14:40 and 17:00 UTC, GitHub’s code search service was fully unavailable; 100% of search queries failed. Service was restored in a degraded state by 17:00 UTC with temporarily stale results, and fully recovered with current data by 23:45 UTC. After initial recovery, search results did not reflect repository changes made after approximately 07:00 UTC that day.
Root Cause: During a routine infrastructure upgrade to the messaging system supporting code search, an automated change was applied too aggressively, causing a coordination failure between internal services. This halted search indexing and search results began going stale. While the team worked to recover the messaging infrastructure, an unintended service deployment cleared internal routing state, escalating the staleness issue into a complete outage.
Remediation: GitHub restored the messaging infrastructure through a controlled restart, reestablishing coordination between services, then reset the search index to a point in time before the disruption. No repository data was lost—the search index is a secondary index derived from Git repositories, which were completely unaffected. GitHub is adding more gradual upgrades with better health checks, deployment safeguards to prevent unintended changes during active incidents, faster recovery tooling, and better traffic isolation.
Incident 2: April 01, 16:06 UTC (4 minutes)
Between 15:34 and 16:02 UTC, the audit log service lost connectivity to its backing data store due to a failed credential rotation. During this 28-minute window, audit log history was unavailable via both the API and web UI, resulting in 5xx errors for 4,297 API actors and 127 github.com users. Events created during this window were delayed by up to 29 minutes. No audit log events were lost. Customers using GitHub Enterprise Cloud with data residency were not impacted.
Root Cause: A failed credential rotation caused the audit log service to lose connectivity to its backing data store.
Remediation: GitHub was alerted to the infrastructure failure at 15:40 UTC—six minutes after onset—and resolved the issue by recycling the affected environment, restoring full service by 16:02 UTC. The team strengthened the credential rotation process and enhanced monitoring configuration, including making paging thresholds more sensitive.
Incident 3: April 09, 09:50 UTC (25 minutes)
Incident 4: April 09, 16:20 UTC (4 hours 16 minutes)
Across two related incidents between 09:05–19:05 UTC and 16:05–20:36 UTC, the Copilot coding agent service was degraded and users experienced significant delays starting new agent sessions. Approximately 84% of new agent session requests were delayed across four separate outage waves, with queue wait times peaking at 54 minutes compared to a normal baseline of 15–40 seconds. The average error rate was 83.9% and peaked at 97.5%. Approximately 22,700 workflow creations were delayed or failed.
Root Cause: A bug in GitHub’s rate limiting logic incorrectly applied a rate limit globally across all users, rather than scoping it to the individual installation that triggered the limit. A contributing factor was a surge in API traffic from a client update that increased requests to an internal endpoint by 3–4x, accelerating rate limit exhaustion. The second incident was also caused by an internal service exceeding API rate limits, compounded by a caching bug that persisted the rate-limited state beyond the actual rate limit window.
Remediation: GitHub detected the issue within 15 minutes and mitigated by disabling the faulty rate limit caching mechanism via feature flag and updating the service to use per-installation credentials, ensuring rate limits are correctly scoped. Service was fully restored by 20:36 UTC. GitHub added automated monitoring and alerting for this failure mode, deployed fixes to reduce unnecessary API traffic through caching improvements, and is continuing work to further isolate rate limit scoping across client types.
Incident 5: April 13, 19:56 UTC (39 minutes)
Between 18:53 and 20:30 UTC, the GitHub Pages service experienced elevated error rates, averaging 10.58% and peaking at 12.77% of requests, resulting in approximately 17.5 million failed requests returning HTTP 500 errors.
Root Cause: An automated DNS management tool erroneously deleted a DNS record for a GitHub Pages backend storage host after its upstream data source intermittently failed to return the record, causing the tool to treat it as stale and remove it. As cached copies of the record expired, GitHub Pages servers could no longer reach the affected storage host.
Remediation: GitHub traced the issue to the missing DNS record and re-created it; service returned to normal by 20:30 UTC. Detection took approximately 53 minutes due to the gradual nature of the error increase and a gap in alerting for this failure type. GitHub is implementing availability-zone-tolerant routing in the Pages frontend so an unresolvable backend host triggers failover to healthy hosts, adding safeguards against automated deletion of DNS records owned by other systems, and improving logging and alerting for DNS resolution failures.
Incident 6: April 16, 15:06 UTC (3 hours 22 minutes)
Between 09:30 and 17:15 UTC, users experienced failures connecting to GitHub Codespaces via the VS Code editor; approximately 40% of codespace start operations failed. Users connecting via SSH were not impacted.
Root Cause: Failures in an upstream service prevented the VS Code Server from being retrieved during codespace startup.
Remediation: GitHub implemented a workaround to use an alternative download path when the primary endpoint is degraded, and coordinated with the upstream dependency team to address the root cause of the download failure. GitHub is improving its fallback mechanism and streamlining processes to accelerate deployment of similar changes.
Incident 7: April 20, 13:28 UTC (15 hours 36 minutes)
Between 10:28 and 15:04 UTC, GitHub experienced degraded service for code scanning default setup, code quality, and project boards; repair of affected project boards additionally lasted until 05:04 UTC on April 21. Code scanning default setup and code quality analyses were not triggered on newly opened pull requests, and newly created issues were not appearing on project boards.
Root Cause: A serialization error prevented proper triggering of code scanning, code quality analyses, and project board updates.
Remediation: GitHub identified the issue within approximately 40 minutes and mitigated it by deploying a fix restoring event publishing for code scanning and code quality; for project boards, an additional code change updated event consumers, followed by a reindex of affected project items. GitHub is strengthening schema validations, improving monitoring for drops in publishing on critical topics, and auditing other parts of the system for similar limitations.
Incident 8: April 22, 15:35 UTC (3 hours 43 minutes)
Between 15:16 and 19:18 UTC, users experienced errors interacting with Copilot Chat on github.com and Copilot Cloud Agent, unable to use either. Copilot Memory (in preview) was also unavailable to Copilot agent sessions.
Root Cause: An infrastructure configuration change resulted in connectivity issues with GitHub’s databases.
Remediation: GitHub identified the cause and restored connectivity to the database. Copilot Chat and Cloud Agent for github.com were restored by 18:16 UTC; remaining regional deployments were restored incrementally, with full resolution at 19:18 UTC. GitHub has taken steps to prevent similar infrastructure changes from causing this kind of database disruption in the future.
Incident 9: April 23, 16:12 UTC (1 hour 18 minutes)
Between 16:03 and 17:30 UTC, users experienced elevated error rates and degraded performance across GitHub Copilot, Webhooks, Git Operations, GitHub Actions, Migrations, and Deployments. Approximately 5–7% of overall traffic was affected. For Copilot, ~7% of AI model requests failed, ~10% of Copilot cloud agent sessions were affected, and ~9% of Copilot Insights dashboard requests returned errors. Webhooks saw ~0.35% of API requests return errors at peak, with up to 10% of traffic experiencing elevated latency (>3 seconds). Git Operations averaged 1.25% errors (peak 2.07%). GitHub Actions workflow run status updates were delayed up to ~8 seconds. For Migrations, 0.88% of active repository migrations failed and 79% saw elevated durations. Deployments were temporarily blocked.
Root Cause: GitHub’s DNS infrastructure in one datacenter entered a degraded state and began intermittently failing to resolve service addresses, causing a cascading impact on services that depend on name resolution to communicate with internal APIs, external providers, and storage systems. The root cause was a recently introduced traffic-balancing mechanism, rolled out progressively to support growth, that under a specific load pattern caused DNS resolvers to begin failing. Existing DNS caching provided partial protection, limiting overall impact to approximately 5–7% of traffic rather than a complete outage.
Remediation: After an initial configuration rollback did not resolve the issue, GitHub restarted the affected DNS infrastructure; services began recovering within minutes and all returned to normal by 17:30 UTC. No data was lost. GitHub is improving DNS infrastructure resilience to prevent single-datacenter failures from cascading, implementing safer rollout and validation with a dedicated environment to test infrastructure changes against production-like traffic, investing in faster automated detection and recovery with self-healing mechanisms for DNS resolution failures, and reducing blast radius by reviewing service dependencies on shared infrastructure components.
Incident 10: April 27, 16:31 UTC (6 hours 15 minutes)
Between 16:15 and 22:46 UTC, GitHub search services experienced degraded connectivity due to saturation of the load balancing tier deployed in front of search infrastructure, causing intermittent failures for services relying on search data including Issues, Pull Requests, Projects, Repositories, Actions, Package Registry, and Dependabot Alerts. Impact varied by search target, with up to 65% of searches timing out or returning an error between 16:15 and 18:00 UTC.
Root Cause: The saturation was caused by a large influx of anonymous distributed scraping traffic crafted to avoid GitHub’s public API rate limits. This scraping traffic made up 30% of the day’s total search traffic, concentrated within a four-hour period, and originated from over 600,000 unique IP addresses with matching actor information in all requests. Existing monitoring did not classify the increased scraping as a risk; this dimension of the incident was only discovered while working to mitigate it.
Remediation: GitHub detected the drop in search results through ongoing monitoring and declared an incident at 16:21 UTC, tracked as mitigated by 21:33 UTC and resolved by 22:46 UTC. GitHub relieved pressure from the load balancers while scaling the load balancing tier, blocking the anomalous traffic, and tuning the balancers. GitHub has since scaled the load balancer tier, applied optimizations to improve connection handling and re-use, and added new monitors and controls to restrict anonymous traffic and mitigate impact to registered users.