GitHub introduced a new monthly reporting practice with this post, publishing on the first Wednesday of each month going forward to detail service incidents and engineering improvements. As the company explained, “our belief is that we can collectively grow as an industry by learning from one another.” The inaugural report covers four incidents from May and June 2020.
May 5 Incident (00:45 UTC, lasting two hours and 24 minutes)
Root Cause: “A shared database table’s auto-incrementing ID column exceeded the size that can be represented by the MySQL Integer type” (2,147,483,647). When larger values were inserted, the system generated 500 errors affecting GitHub Apps, including Actions, Pages, and Dependabot.
Prevention: GitHub committed to “extending our test frameworks to include a linter in place for int / bigint foreign key mismatches.”
May 22 Incident (16:41 UTC, lasting five hours and nine minutes)
Root Cause: During scheduled MySQL failover maintenance, “a novel crash in the mysqld process on the newly promoted MySQL primary server” occurred. It took approximately four hours to restore replicas and another hour to reconfigure the cluster, causing visible delays for users before data appeared in interfaces.
Prevention: GitHub committed to “run multiple internal gameday exercises in response to ensure a higher degree of preparedness.”
June 19 Incident (08:52 UTC, lasting 51 minutes)
Root Cause: A/B testing instrumentation introduced an undocumented dependency on a dynamically generated file. During deployment, the “file failed to be generated on a significant proportion of the application deployments due to a high retrieval rate being rate limited,” causing site-wide errors.
Remediation: The team disabled the file requirement, and “configuration for A/B and multivariate experiments will be cached internally.”
June 29 Incident (12:03 UTC, lasting two hours and 29 minutes)
Root Cause: Following a ProxySQL update from June 22, “the newly promoted primary crashed. Orchestrator’s anti-flapping mechanism prevented a subsequent automatic failover,” and multiple MySQL primary crashes followed.
Remediation: GitHub “rolled back to the previous version of ProxySQL and disabled a change in our application” to resolve the underlying CPU starvation issues.