In March, GitHub experienced four incidents that resulted in degraded performance across GitHub services.
Incident 1: March 03, 18:59 UTC (1 hour 10 minutes)
Between 18:46 and 20:09 UTC, GitHub experienced a period of degraded availability impacting github.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other dependent services. At the peak of the incident, github.com request failures reached approximately 40%. Approximately 43% of GitHub API requests failed. Git operations over HTTP had an error rate of approximately 6%, while SSH was not impacted. GitHub Copilot requests had an error rate of approximately 21%. GitHub Actions experienced less than 1% impact.
Root Cause: This incident shared the same underlying cause as an incident in early February: a large volume of writes to the user settings caching mechanism. While deploying a change to reduce the burden of these writes, a bug caused every user’s cache to expire, get recalculated, and get rewritten. The increased load caused replication delays that cascaded down to all affected services.
Remediation: GitHub mitigated the issue by immediately rolling back the faulty deployment. The company added a killswitch and improved monitoring to the caching mechanism to ensure it is notified before there is user impact and can respond swiftly, and is moving the cache mechanism to a dedicated host, ensuring future issues will solely affect services that rely on it.
Incident 2: March 05, 16:35 UTC (2 hours 55 minutes)
Between 16:24 and 19:30 UTC, GitHub Actions was degraded. During this time, 95% of workflow runs failed to start within 5 minutes with an average delay of 30 minutes, and 10% of workflow runs failed with an infrastructure error.
Root Cause: Redis infrastructure updates being rolled out to production to improve resiliency introduced a set of incorrect configuration changes into GitHub’s Redis load balancer, causing internal traffic to be routed to an incorrect host.
Remediation: GitHub mitigated the incident by correcting the misconfigured load balancer; Actions jobs were running successfully starting at 17:24 UTC. GitHub immediately rolled back the updates that were a contributing factor, froze all changes in this area until follow-up work was complete, and is improving automation to ensure incorrect configuration changes cannot propagate through its infrastructure, improving alerting to catch misconfigured load balancers before they cause an incident, and updating Redis client configuration in Actions to improve resiliency to brief cache interruptions.
Incident 3: March 19–20, 13:44 UTC (48 minutes)
Between 01:05 and 02:52 UTC on March 19, and again between 00:42 and 01:58 UTC on March 20, the Copilot Coding Agent service was degraded and users were unable to start new Copilot Agent sessions or view existing ones. During the first incident, the average error rate was approximately 53% and peaked at approximately 93% of requests. During the second incident, the average error rate was approximately 99% and peaked at approximately 100% of requests, with significant retry amplification.
Root Cause: Both incidents were caused by the same underlying system authentication issue that prevented the service from connecting to its backing datastore. The second occurrence was due to an incomplete remediation of the first.
Remediation: GitHub mitigated each incident by rotating the affected credentials, which restored connectivity and returned error rates to normal. The company has implemented automated monitoring for credential lifecycle events and is improving operational processes to reduce time to detection and mitigation of similar issues.
Incident 4: March 24, 16:59 UTC (2 hours 52 minutes)
Between 15:57 and 19:51 UTC, the Microsoft Teams Integration and Teams Copilot Integration services were degraded and unable to deliver GitHub event notifications to Microsoft Teams. On average, the error rate was 37.4% and peaked at 90.1% of requests; approximately 19% of all integration installs failed to receive GitHub-to-Teams notifications during this period.
Root Cause: An outage at one of GitHub’s upstream dependencies, which caused HTTP 500 errors and connection resets for the Teams integration.
Remediation: GitHub coordinated with the relevant service teams, and the issue was resolved at 19:51 UTC when the upstream incident was mitigated. GitHub is working to update observability and runbooks to reduce time to mitigation for similar issues in the future.