In December, GitHub experienced three incidents that resulted in degraded performance across services.
Incident 1: Codespace Creation Failures & Resume Issues — December 27 (02:30 UTC, lasting 90 minutes)
Root Cause: A bug was triggered during HMAC secret rotation between GitHub’s frontend and internal services. When a key was disabled in Azure Key Vault while rolling back the rotation, API calls between the two services started failing, causing codespaces and background functions to malfunction.
Impact: All codespace creations failed; approximately 15% of resumes failed; other background functions were affected.
Resolution: The team temporarily re-enabled the key in Key Vault, then deployed a fix to continue the secret rotation.
Preventive Measures: Playbooks and Azure Key Vault documentation were improved.
Incident 2: Email Notification Failures — December 28 (05:52 UTC, lasting 65 minutes)
Root Cause: Authentication credential rotation between backend services and SMTP servers left some servers with outdated credentials, causing failed authentication between backend services that generate notifications and a subset of SMTP servers.
Impact: CI activity and Gist email notifications were primarily affected.
Resolution: Engineers updated SMTP server credentials.
Preventive Measures: Secrets rotation playbooks and alerting systems were enhanced to provide earlier detection.
Incident 3: Sign-In/Sign-Up Outage — December 29 (00:34 UTC, lasting 68 minutes)
Root Cause: Credential rotation wasn’t reflected in frontend caches, creating a mismatch in behavior between signed in and signed out users.
Impact: Users were unable to sign in or create accounts; existing sessions were unaffected.
Resolution: Updated credentials were deployed to cache services.
Preventive Measures: Monitoring improvements are underway.