Incident Report: Dec 13th, 2023

Author: Angelo Saraceno Date: Dec 13, 2023

Summary

Railway experienced a deployment outage from 21:45 UTC to 23:24 UTC on December 13th, 2023. The incident stemmed from “a large sagging under load” on Google’s Metadata server during a routine GKE cluster upgrade, which “caused encrypt and decrypt requests to become severely delayed.”

Incident Details

The root cause involved a known Google issue affecting GKE versions 1.25-1.27. During the standard v1.25 rollout at 08:13 UTC, Railway’s engineering team discovered that “the metadata server pods were updated to their latest version, which had a documented issue at high load.”

The severity escalated when, at 20:57 UTC, Railway sent migration notices to users, creating “a spike in load as users who underwent the migration increased the load on the metadata service.”

Impact

New deployments were affected, as “deployments would not have the most current secrets.” Users “were prevented from updating any environment variable,” disrupting routine platform maintenance. However, existing workloads remained unaffected.

Response and Resolution

Railway engineering deployed a newer GKE version at 22:59 UTC. By 23:10 UTC, variable decryption succeeded, and full service restoration was confirmed by 23:24 UTC.

Preventative Measures

Railway committed to three actions: unifying incident response channels, improving migration communication timing, and accelerating migration away from Google Cloud infrastructure.