In September, GitHub experienced two incidents that resulted in degraded performance across GitHub services, along with follow-up detail on two August incidents affecting Codespaces and Actions.
September 8 Incident (19:44 UTC, lasting 5 hours and 11 minutes)
Monitoring detected an increase in the number of pull request merge failures. The issue primarily impacted Enterprise Managed Users (EMUs), with some bot accounts also affected.
Root Cause: A data transition that removed inconsistent data from profile records inadvertently operated on EMU accounts, removing some data that is required to successfully merge pull requests via the UI and API. CLI merges continued to function normally.
Resolution: The team restored affected data from backup, which took longer than anticipated. They considered a code workaround but rejected it to avoid introducing further inconsistencies. Recovery was complete by September 9 at 00:55 UTC.
Follow-up Actions: Modified data transition procedures for faster restores and automatic rollback capabilities; enhanced testing processes for EMUs.
September 28 Incident (03:53 UTC, lasting 1 hour and 16 minutes)
Alerts indicated an incident that impacted most Codespaces customers. Investigation was still ongoing at the time of publication; GitHub said it would provide more detail in the October report.
Follow-up: August 29 Incident (12:51 UTC, lasting 5 hours and 40 minutes)
Monitoring detected increased Codespaces create and start errors alongside DNS-related networking failures in running instances.
Root Cause: An Ubuntu security patch in systemd broke DNS resolution. Codespaces host VMs had unattended upgrades for security fixes enabled by default, causing the patch to be applied automatically to running systems.
Resolution: GitHub disabled systemd’s DNS resolver and manually configured upstream addresses at 18:13 UTC; recovery was confirmed by 18:31 UTC.
Follow-up Actions: Updating DNS configuration to reduce dependency on systemd; evaluating whether to disable unattended upgrades to enable more deterministic behavior.
Follow-up: August 18 Incident (14:33 UTC, lasting 3 hours and 23 minutes)
A sudden spike in traffic to GitHub Actions created higher than usual write load, causing an internal token-generation service to return 503 errors and hit database connection limits.
Root Cause: A mismatch between client-side throttling limits and database capacity allowed excessive traffic beyond what the system could handle.
Resolution: GitHub scaled the affected database and increased concurrent connection limits; service recovered by 17:36 UTC.
Follow-up Actions: Enhanced monitoring and alerting; evaluating improvements to the throttling mechanism.