In December, GitHub experienced two incidents affecting service availability.
December 17 Incident (14:33 UTC, lasting 17 minutes)
Between 14:33 and 14:50 UTC, GitHub experienced broad service disruption affecting login, repository viewing, pull requests, and issue commenting, with error rates averaging 8.5% and peaking at 44.3%.
Root Cause: Planned maintenance unintentionally caused the live updates service to fail. With live updates unavailable, clients refreshed aggressively, further overloading GitHub’s servers.
Resolution: The team rolled back the changes from the planned maintenance and scaled up the service to handle the influx of traffic from WebSocket clients.
Preventive Measures: GitHub is adding monitoring higher in the request path to capture this type of failure, improving alerting to better detect the scope of impact from incidents, and working to reduce the impact of the live updates service’s availability on github.com.
December 20 Incident (15:57 UTC, lasting 43 minutes)
Between 15:57 and 16:39 UTC, users encountered 500 errors on GitHub’s marketing pages, with no impact to any operational product or service area.
Root Cause: A partial outage of one of GitHub’s third-party service providers.
Resolution: The service provider resolved the outage at 16:39 UTC.
Preventive Measures: GitHub is investigating methods to improve error handling and gracefully degrade these pages in case of future outages.