In August, GitHub experienced three incidents affecting service availability.
August 5 Incident (15:33 UTC, primary issue lasting 32 minutes; secondary issue lasting until 19:45 UTC)
Between 15:33 and 16:13 UTC, pushes, webhooks, notifications, and pull requests were affected, along with approximately 4% of web and REST API traffic.
Root Cause: A production database migration dropped a column from a pull request functionality table. The ORM continued to reference the dropped column in a subset of pull request queries, causing elevated error rates.
Resolution: GitHub deployed a change instructing the ORM to ignore the removed column. However, the fix applied only to the largest production environment; custom and canary environments missed the update, triggering a secondary incident affecting approximately 0.1% of pull request traffic until 19:45 UTC.
Preventive Measures: GitHub is adding application monitoring to prevent continued rollout when impact is observed, and streamlining changes across environments to prevent similar secondary incidents.
August 12 Incident (13:30 UTC, lasting 3 hours 44 minutes)
Between 13:30 and 17:14 UTC, GitHub search, and the issues, pull requests, projects, and deployments pages, along with Actions workflow and label filters, were degraded. Up to 75% of search queries failed between 14:00-15:30 UTC, and search result updates were delayed up to 100 minutes.
Root Cause: Intermittent connectivity issues between load balancers and search hosts. Retry logic masked the problems initially, but retry queues eventually overwhelmed the load balancers, causing failure.
Resolution: GitHub throttled the search indexing pipeline at 15:30 UTC to reduce load; an automated reboot of a search host at 17:14 UTC resolved the connectivity failures.
Preventive Measures: GitHub improved internal monitors and playbooks, and tuned the search cluster load balancer configuration.
August 27 Incident (20:35 UTC, lasting 46 minutes)
Between 20:35 and 21:17 UTC, Copilot averaged a 36% request failure rate (peaking at 77%), and approximately 2% of non-Copilot web and REST API traffic failed.
Root Cause: A production database migration dropped a column from a Copilot functionality table. The ORM continued to reference the dropped column, generating numerous 5xx responses, similar to the August 5 incident.
Resolution: GitHub applied a fix to the production schema at 21:15 UTC, with full recovery by 21:17 UTC.
Preventive Measures: GitHub implemented a temporary block on all drop column operations, is adding safeguards to prevent future similar issues, and is implementing graceful degradation so Copilot issues won’t impact other product features.