In February, GitHub experienced two incidents affecting service availability.
February 25 Incident (14:25 UTC, lasting 2 hours 19 minutes)
Between 14:25 and 16:44 UTC, email and web notifications were delayed. At the peak of the incident, approximately 10% of all notifications took over 10 minutes to be delivered, with the remaining ~90% delivered within 5-10 minutes.
Root Cause: Worker pools running too close to capacity at peak times, resulting in a delay in queue processing.
Resolution: GitHub scaled out the service to meet demand.
Preventive Measures: GitHub established a higher baseline capacity to ensure no further delays occur, and is improving capacity planning to proactively manage the pool going forward.
February 3 Incident (18:01 UTC, lasting approximately 30 minutes)
Migration Tools experienced a complete outage for all users attempting migrations during the window.
Root Cause: A deployment of a system component led to missing Docker images.
Resolution: GitHub rolled back to the previous stable version, restoring service.
Preventive Measures: GitHub enhanced test coverage and workflows to ensure validation of critical dependencies.