GitHub experienced two significant incidents in April 2021 affecting API requests and the GitHub Packages service, particularly the Container registry.
Incident 1: April 1 (21:30 UTC, lasting one hour and 34 minutes)
Root Cause: “This incident was caused by failures in our DNS resolution, resulting in a degraded state of availability for the GitHub Packages Container registry service.” Internal services supporting the Container registry experienced intermittent connection failures to dependent services, preventing users from completing container operations. Only beta users were affected; the broader Packages service remained operational.
Prevention: “We are looking at increasing the cache times of our DNS resolutions to decrease the impact of intermittent DNS resolution failures in the future.”
Incident 2: April 22 (17:16 UTC, lasting 53 minutes)
Root Cause: “A bug was discovered that broke the ability of the application to successfully write to the secret scanning database.” The migration, intended to isolate secret scanning tables into a separate cluster, exposed an undocumented dependency: repository creation required secret scanning functionality, so repository creation was blocked entirely until the team rolled back the migration.
Remediation/Prevention: GitHub “actively working with our vendor to update the data migration tool,” revised the migration process with improved recovery steps, and “application code has been updated to remove the dependency on secret scanning for the creation of repositories.”