In August, GitHub experienced two incidents that resulted in degraded performance across GitHub services.
Incident 1: Webhook Delivery Delays (August 15, 16:58 UTC, lasting 4 hours and 29 minutes)
GitHub began experiencing increasing delays in an internal job queue processing webhooks. The status page was updated to yellow at 17:24 UTC. Customers experienced webhooks delays as long as 4.5 hours.
Root Cause: The delays were caused by a significant and sustained spike in webhook deliveries that created a backup in the delivery queue.
Resolution: The team mitigated the issue by blocking events from sources of the increased load, which allowed the system to gradually recover as the backlog of events was processed.
Follow-up Actions: Improvements were implemented to handle higher traffic volumes and manage load sources more effectively.
Incident 2: Background Job Processing Failure (August 29, 02:36 UTC, lasting 49 minutes)
GitHub systems experienced widespread delays in background job processing, affecting webhook deliveries, GitHub Actions, and asynchronous workloads. Workloads were delayed by up to an hour, but no data was lost.
Root Cause: A job queueing service component failed due to an interaction with unexpected CPU throttling and short session timeouts for a Kafka consumer group, causing the consumer to become stuck in a restart loop.
Resolution: The team mitigated the issue by shifting the load to the standby service as well as redeploying the primary service.
Follow-up Actions: Enhanced monitoring and preventive measures were planned.