GitHub experienced two incidents in November 2022 causing service degradation, plus follow-up detail on an October incident affecting Codespaces.
November 25 Incident (16:34 UTC, lasting 1 hour and 56 minutes)
GitHub Packages and Pages experienced degraded performance. Alerting systems detected performance issues affecting customers using these services.
Root Cause: Investigation was ongoing at the time of publication. GitHub said it would provide a more detailed update on cause and remediation in the January Availability Report.
Follow-up: October 26 Incident (00:47 UTC, lasting 3 hours and 47 minutes)
Customers could not create or resume Codespaces in the East US region. Manual region switching was offered for new instances, though resuming in alternate regions was unavailable.
Root Cause: A third-party database provider experienced an outage. GitHub’s service lacked regional data replication, which prevented immediate communication with the East US database.
Resolution/Follow-up Actions: Regional replication and failover mechanisms were implemented to mitigate future incidents.
November 3 Incident (16:10 UTC, lasting 1 hour and 2 minutes)
Customers experienced high latency in receiving webhook deliveries and in starting GitHub Actions workflow runs, along with error responses affecting Actions, webhooks, the API, repositories, pull requests, and Codespaces. The majority of repositories attempting workflows faced delays exceeding five minutes.
Root Cause: An inefficient database query checking repository security advisory access, triggered by poorly optimized API calls, created a webhook queue backup that affected background jobs.
Resolution: GitHub fixed the inefficient query.
Follow-up Actions: Reviewed rate limits, increased webhooks queue throttling, and investigated reducing background job impacts on Actions workflows.