In July, GitHub experienced a single significant incident.
July 13 Incident (08:18 UTC, lasting 4 hours 25 minutes)
Root Cause: The outage began when production Kubernetes Pods became unavailable because “a single container within the Pod was exceeding its defined memory limits and being terminated.” The container’s ImagePullPolicy was configured to “Always,” requiring a fresh image fetch on every restart. However, “due to a routine DNS maintenance operation that had been completed earlier, our clusters were unable to successfully reach our registry resulting in Pods failing to start.” This triggered a cascading failure across clusters, reducing overall capacity, and remediation attempts through redeployment propagated the failure further across production infrastructure.
Resolution: “It wasn’t until we restarted the process with the cached DNS records that we were able to successfully fetch container images, redeploy, and recover our services.”
Preventive Measures: GitHub committed to “Enhancing monitoring,” “Minimizing our dependency on the image registry,” “Expanding validation during DNS changes,” and “Reevaluating all the existing Kubernetes deployment policies,” alongside implementing progressive deployment improvements for incremental impact assessment.