In August, GitHub experienced a new Codespaces outage, plus a detailed follow-up on a previously reported July incident.
Incident 1: August 29 Codespaces Outage (12:51 UTC, lasting 5 hours 40 minutes)
Alerting systems detected an incident affecting most Codespaces customers with significant impact and degraded availability. Root cause was still under investigation at time of publication; GitHub noted: “Due to the recency of this incident, we are still investigating the contributing factors,” and promised detailed findings in the September report.
Incident 2: July 27 Codespaces Creation Failures (Follow-up) (22:29 UTC, failures began ~21:30 UTC, lasting 7 hours 55 minutes)
Codespaces in East US and West US regions using 2-core and 4-core machine types could not be created or restarted. Many requests timed out and failed.
Root Cause: A cloud provider update in those regions was “incompatible with the way we built our host VM image.” The provider’s update caused VM creation failures that went undetected because “the VMs were being created successfully even though some critical processes…were not running.”
Remediation:
- Adjusted the image generalization process per provider recommendation
- Scaled VM pools to absorb high failure rates
- Modified timeouts to accelerate failure detection
- Audited the VM image building process for future prevention
- Improved service logic and monitoring to verify critical operations execute during VM creation
- Enhanced alerting to detect failures earlier