In October, GitHub experienced four incidents that resulted in significant impact and degraded state of availability to multiple GitHub services, plus follow-up detail on a September Codespaces incident.
October 26 Incident (00:47 UTC, lasting 3 hours and 47 minutes) — Codespaces Outage
Alerting systems detected an incident impacting most Codespaces customers. Root cause was still under investigation at the time of publication; GitHub said detailed findings would follow in the November report.
October 13 Incident (20:43 UTC, lasting 48 minutes) — Projects API Error
An increase was detected in the Projects API error response rate; the Issues service moved to red status at 20:47 UTC.
Root Cause: A change introduced a database validation that required a certain value to be present, but it did not correctly set a default value in every scenario. This resulted in the population of null values in some cases, which produced an error when pulling certain records.
Resolution: Rollback initiated at 21:08 UTC; status improved at 21:24 UTC (yellow) and was fully resolved at 21:31 UTC.
Follow-up Actions: Added mitigations against missing values, improved testing procedures, and fixed deployment dashboards that contained inaccurate pre-production error data.
October 12 Incident (23:27 UTC, lasting 3 hours and 31 minutes) — Codespaces Configuration Change
A global configuration change rolled out at 22:30 UTC; new Codespace creation declined significantly by 23:15 UTC, and the service moved to yellow then red status.
Root Cause: One of the older components of the backend system did not cope well with the configuration change, causing a schema conflict. This was not properly tested prior to rollout. Additionally, this component version did not support gradual exposure across regions, so many regions were impacted at once.
Resolution: GitHub carefully rolled back the schema change, a complex procedure requiring extended time; service returned to green at 02:58 UTC.
Follow-up Actions: Eliminating the dependency on the older configuration type; future changes must follow safe deployment practices with gradual regional rollout.
October 5 Incident (06:30 UTC, lasting 31 minutes) — Webhooks Backlog
Webhooks experienced a severe backlog from high-volume automated activity causing rapid create/delete operations. Exceptions occurred when webhook payloads referenced deleted database data, tying up workers and preventing new event processing. GitHub Actions moved to red status.
Root Cause: Automation creating and deleting many repositories in quick succession, combined with webhook delivery workers unable to process events when the source data no longer existed.
Resolution: GitHub disabled the automated accounts causing the activity, updated webhook delivery workers to skip jobs lacking required database data, then re-enabled the automated accounts once the fix was deployed.
Follow-up Actions: Committed to improving service resilience to load spikes.
Follow-up: September 28 Incident (03:53 UTC, lasting 1 hour and 16 minutes) — Codespaces Secret Rotation
Port forwarding failed on the Codespaces web client; the service moved to yellow status at 03:53 UTC. Later the same day, Codespaces creation/start failures increased across all regions.
Root Cause: A missed step in the secret rotation checklist a few hours earlier caused some downstream components to fail to pick up the new secret. The system didn’t automatically pick up the new secret as expected and required a restart. Secret exchange token caching (up to 24 hours) delayed detection in a downstream component.
Resolution: GitHub ran the missed rotation step at 04:29 UTC, restoring port forwarding. After identifying a monitoring gap (alerts tracked error rates but not traffic anomalies), the team restarted the service in all regions at 18:27 UTC, increased pool size at 18:45 UTC to process the backlog, and located and rotated the missed instance in West Europe at 19:44 UTC.
Follow-up Actions: Updated the secret rotation checklist with the missing steps, added verification steps for secret propagation, is automating secret rotation processes to reduce human error, and added monitoring to track secret versions in use across components.