In April, GitHub experienced three incidents affecting Codespaces, an unresolved multi-service disruption, and GitHub Packages.
Incident 1: April 1, 7:07 UTC (lasting 5 hours 32 minutes)
Codespaces service experienced widespread creation and startup failures across multiple regions.
Root Cause: Expired secrets used by the Codespaces service caused resource pools to deplete, initially in US West. The affected pools had no backup capacity when dependent services failed. Communication delays prevented timely credential rotation across a limited group of authorized engineers.
Remediation:
- Refreshed and rolled out expired credentials to all regions
- Implemented advance monitoring for resource expiration
- Added alerts for resource pool threshold drops
- Initiated migration to eliminate secret-based authentication mechanisms
Incident 2: April 14, 20:35 UTC (lasting 4 hours 53 minutes)
GitHub stated the team was still investigating contributing factors, with details promised in the May Availability Report.
Incident 3: April 25, 8:59 UTC (lasting 5 hours 8 minutes)
High CPU utilization on the GitHub Packages Registry database caused internal server errors, preventing package push/pull operations and UI access.
Root Cause: Unexpected high-volume “Create Manifest” command usage exposed a performance bug. Database-level throttling proved insufficient to contain the anomalous activity.
Remediation:
- Blocked the problematic activity to restore service
- Improved rate limiting for package operations
- Fixed the underlying performance bug
- Enhanced database alerting thresholds for faster detection
- Decoupled package listing from the repository homepage to prevent cascading failures