In February, GitHub experienced two incidents that resulted in degraded performance across services.
Incident 1: February 26, 2024 (18:34 UTC, lasting 53 minutes)
Root Cause: Capacity constraints with the job queuing service and a failure of the automated failover system.
Impact: Webhooks, GitHub Actions, and UI updates (such as pull request notifications) were affected.
Resolution: Manual failover to a secondary cluster; no data loss occurred.
Incident 2: February 29, 2024 (09:32 UTC, lasting 142 minutes)
Root Cause: Processing delays stemming from background job service issues; improper restoration to the primary system at 10:32 UTC caused a backlog.
Impact: Webhooks, GitHub Actions, and GitHub Issues were affected, with 95% of delays occurring between 11:05–11:27 UTC.
Resolution: A correction was made at 11:21 UTC, with full restoration at 11:27 UTC.
Preventive Measures: GitHub implemented three short-term improvements — better automation, increased failover reliability, and expanded background job queuing capacity — with longer-term work focused on improving overall scalability and reliability of the job processing platform.