GitHub experienced three significant incidents in March 2021 affecting issues, pull requests, webhooks, API requests, GitHub Pages, and GitHub Actions services.
Incident 1: March 1 (09:59 UTC, lasting 1 hour 42 minutes)
Root Cause: Check suite IDs exceeded the maximum Int32 value. Although the database had been migrated to bigint six months prior, “a GraphQL library we depend on” was using Int32 when unmarshalling JSON. This caused Actions jobs to fail to queue properly, and workflow search functionality returned incomplete results.
Remediation: “We deployed a code fix to mitigate after validating it would not lead to data integrity issues.” The search index required rebuilding to restore search completeness.
Prevention: GitHub “audited and updated usage of all external libraries” and conducted “drill testing exercises” simulating check run ID overflow scenarios.
Incident 2: March 12 (19:11 UTC, lasting 1 hour 10 minutes)
Root Cause: “A database migration to flip the order of an index” was intended to optimize query performance but caused “a full table scan since there was a missed dependency on the changed index,” degrading availability across GitHub.com with cascading timeout issues across dependent services.
Remediation: The index was restored.
Prevention: GitHub is “determining better tooling to identify index regressions” and “created an inventory of indexes used by generated queries.”
Incident 3: March 15 (20:38 UTC, lasting 1 hour 18 minutes)
Root Cause: “An outage from our authentication provider that is used for our infrastructure” prevented hosted job infrastructure access. All hosted Actions jobs were queued indefinitely, and build requests delayed over 30 minutes were abandoned.
Prevention: GitHub is “determining ways to extend the lifetime of authentication tokens to handle brief outages.”