In October, GitHub experienced four incidents affecting service availability.
October 9 Incident (14:35 UTC, lasting 1 hour 46 minutes)
Between 14:35 and 15:21 UTC, GitHub’s UI, API, Actions, and Large File Storage were affected. UI users saw increased latency during the first 5 minutes; the API had error rates up to 7.3% before stabilizing to 0.05%; 24% of Actions runs were delayed by an average of 13 minutes; and 0.038% of LFS requests errored.
Root Cause: A network device in maintenance mode that was undergoing repairs was brought back into production before repairs were fully completed.
Preventive Measures: GitHub is enhancing the validation process for device repairs of this category.
October 17 Incident (12:51 UTC, lasting 1 hour 10 minutes)
Between 12:51 and 14:01 UTC, mobile push notifications for github.com and GitHub Enterprise Cloud experienced complete notification delivery failure across all regions.
Root Cause: An erroneous configuration change to cloud resources used for mobile push notification delivery.
Preventive Measures: GitHub is reviewing procedures and management of these cloud resources.
October 20 Incident (08:05 UTC, lasting 2 hours 45 minutes)
Between 08:05 and 10:50 UTC, Codespaces creation had a 39.5% average error rate (peaking at 71%), and resume operations had a 23.4% average error rate (peaking at 46%).
Root Cause: A cascading failure triggered by an outage in a third-party dependency required to build devcontainer images.
Preventive Measures: GitHub is evaluating options to remove the dependency from the critical path and improving monitoring.
October 29 Incident (14:07 UTC, lasting 9 hours 8 minutes)
Between 14:07 and 23:15 UTC, Codespaces (90% average error rate, 100% peak), Actions larger hosted runners (0.5% of workflows and 10% of jobs failed or delayed), GitHub Enterprise Importer (migration failures during git push), and the Copilot Metrics API (approximately 100 failed API requests) were affected.
Root Cause: A widespread outage at a third-party provider.
Resolution: Full recovery was achieved once the external provider resolved its outage.
Preventive Measures: GitHub is reducing critical path dependencies on external providers and implementing strategies to gracefully degrade services.