In August, GitHub experienced two incidents causing degraded service performance.
August 10 Incident (15:16 UTC, lasting 1 hour 17 minutes)
A MySQL database primary entered a degraded state, impacting multiple internal services. Users experienced an inability to perform write operations across Git operations, API requests, webhooks, issues, pull requests, GitHub Pages, GitHub Packages, and GitHub Actions.
Root Cause: An edge case in a frequently-used application generated a poorly performing query capable of impacting overall database capacity. The issue combined with application retry and queueing logic, preventing the MySQL cluster from automatic recovery.
Resolution: The problematic query was addressed, and application retry logic was modified to reduce the likelihood of recurrence.
August 10 Incident (19:57 UTC, lasting 3 hours 6 minutes)
During Actions service maintenance, high error rates emerged on new and in-progress workflow runs, causing all queued jobs to fail temporarily.
Root Cause: Setup work for a new Actions Premium Runner microservice involved altering service discovery processes. A bad service record prevented microservices from making service-to-service calls.
Resolution: The bad record was removed from service discovery infrastructure. Follow-up improvements included fixing service discovery handling and enhancing visibility into Actions microservice changes.