In June, GitHub experienced four incidents plus a follow-up on two previously reported May incidents.
Incident 1: June 1, 09:40 UTC (lasting 48 minutes)
Customers experienced delays in GitHub Actions workflow startups due to excessive load on a database proxy server.
Root Cause: Data analysis queries were “pointed at an incorrect database,” creating substantial load that caused proxy server pods to crash-loop due to out-of-memory errors.
Remediation:
- Increased available memory to proxy pods (completed by 10:08 UTC)
- Moved data analysis queries to “a dedicated analytics setup that does not serve production traffic”
- Added alerts for proxy server load increases
- Investigating guardrails to limit production database access
Incident 2: June 21, 17:02 UTC (lasting 1 hour 10 minutes)
Users with Marketplace or Sponsorship subscription plans couldn’t use Copilot (post-GA). Less than 20% of active users were impacted due to authentication token API errors.
Root Cause: An issue with the API responsible for creating authentication tokens for these subscription types.
Remediation:
- Rolled out an API endpoint fix by 17:45 UTC
- Increased testing for subscription type combinations
- Added scenarios to user testing
- Enhanced data shape testing before future rollouts
Incident 3: June 28, 17:16 UTC (lasting 26 minutes)
Degraded availability was detected for Codespaces. Investigation was ongoing at publication, with a detailed update promised in the July Availability Report.
Incident 4: June 29, 14:48 UTC (lasting 1 hour 27 minutes)
Multiple services were impacted including GitHub Actions, API Requests, Codespaces, Git Operations, GitHub Packages, and GitHub Pages. Investigation was ongoing at publication, with a detailed update and future mitigation efforts promised in the July Availability Report.
Follow-up: May 27 Incidents (two instances)
May 27, 04:26 UTC (lasting 21 minutes) and May 27, 07:36 UTC (lasting 1 hour 21 minutes).
Both instances involved elevated load concentrated on the login endpoint, causing increased error rates across API requests.
Root Cause: Sustained high-traffic load patterns affecting the login endpoint.
Remediation:
- Network-level mitigation deployed
- Application-side mitigations added for sustained loads
- Architectural changes planned for service resilience
- Improved on-call procedures to “ensure we always report the correct status level based on SLO review”