In May, GitHub experienced three incidents plus a follow-up on a previously reported billing incident.
Incident 1: May 20, 09:44 UTC (lasting 49 minutes)
Container registry experienced increased latency and internal server errors affecting package push/pull operations and UI access.
Root Cause: “The throttling criteria configured at the API side for this command was too permissive, and a database query was found to be non-performant under that degree of scale.” A high volume of “Put Manifest” commands caused CPU spikes on the Container registry database.
Resolution: Throttled requests from all organizations/users and restarted front-end servers and the database to restore normal operation.
Future Prevention: Added separate rate limiting by organization/user and committed to SQL query performance improvements.
Incident 2: May 27, 04:26 UTC (lasting 21 minutes)
Alerting systems detected degraded API request availability. Root cause and resolution were stated as still under investigation, with a detailed update promised in the June Availability Report.
Incident 3: May 27, 07:36 UTC (lasting 1 hour 21 minutes)
Multiple services were impacted: GitHub Actions, API Requests, Codespaces, Git Operations, Issues, Packages, Pages, Pull Requests, and Webhooks. Contributing factors were still under investigation, with a detailed update promised in the June report.
Follow-up: April 14 Billing Incident (lasting 4 hours 53 minutes)
Actions and Codespaces customers saw incorrect charges; many hit spending limits, blocking service usage.
Root Cause: “Actions and Codespaces minutes were mistakenly multiplied by 1,000,000,000 to convert gigabytes into bytes when this was not necessary.” The problematic change affected shared metered billing code.
Resolution: Code was reverted; corrupted billing data was repaired over 24 hours before metered billing was re-enabled; customers were given unlimited usage during the incident at no charge.
Future Prevention: Added a Rubocop rule blocking unsafe billing code; implemented anomaly monitoring for billed quantities; tightened the release process to require feature flags and end-to-end tests.