GitHub Availability Report: October 2020

In October, we experienced one incident resulting in significant impact and degraded state of availability for issues, pull requests, webhooks, GitHub Actions, and GitHub Pages services.

October 9 21:30 UTC (lasting for two hours and 32 minutes)

While reprovisioning ZooKeeper nodes as part of routine upgrades, new hosts were introduced too quickly, which resulted in the election of a second leader, effectively introducing a logically distinct second ZooKeeper cluster where there should have been only one.

While the ZooKeeper hosts were in this state, a single Kafka broker in the cluster that powers the internal background job system connected to the newly-formed second ZooKeeper cluster and elected itself as the Kafka controller. At this point, there were two distinct Kafka clusters serving conflicting cluster state information to clients. This incorrect state caused write failures for approximately 10% of requests to the background job service, resulting in a backup of jobs as traffic and worker capacity migrated to the secondary job processing system.

No background jobs were lost during this incident. While significant queue backups occurred for some systems, retry behavior in clients and redundant queueing systems mitigated such issues.

To avoid this class of failure in the future, the team updated the ZooKeeper provisioning checklist and plans to introduce automation for ZooKeeper and Kafka cluster maintenance.