In June, GitHub experienced two incidents that resulted in degraded performance across GitHub services.
Incident 1: June 7 Incident (16:11 UTC, lasting 2 hours and 28 minutes)
GitHub encountered escalating delays in an internal job queue responsible for processing Git pushes. Monitoring systems triggered alerts after 19 minutes. During this event, customers experienced GitHub Actions workflow and webhook delays reaching up to 55 minutes. Pull requests failed to accurately display new commits.
Root Cause: A customer executed a substantial number of pushes to a repository with particular data characteristics. These jobs became throttled during Git backend communication, extending execution times. The slow jobs consumed available worker capacity, preventing other repositories’ push processing.
Resolution: Once identified, the problematic source was temporarily disabled. The system gradually recovered as the job backlog cleared.
Follow-up Actions: Modified Git backend throttling to fail more rapidly and decreased the Git client timeout within jobs to prevent hanging. Additional improvements were planned to accelerate detection, diagnosis, and recovery.
Incident 2: June 29 Incident (14:50 UTC, lasting 32 minutes)
Starting at 17:39 UTC, GitHub experienced outages affecting portions of North America, particularly the US East Coast and South America, lasting approximately 32 minutes.
Root Cause: GitHub had constructed a secondary internet edge facility (completed in January) operating in high availability alongside the primary facility. During network redundancy validation testing, a live failover test inadvertently triggered a production outage. The test revealed a network path configuration defect in the secondary facility preventing proper primary operation.
Resolution: Within two minutes of alerting, the team reverted the change and restored the primary facility. Traffic rebalancing and border router reconvergence took additional time to fully restore public connectivity.
Follow-up Actions: The configuration issue was resolved, and gaps in both configuration and failover testing procedures are being addressed to enhance system resilience.