Incident Report: December 16th, 2024
Published by Jacob Cooper on Dec 16, 2024
Overview
Railway experienced an outage affecting inbound traffic on Google Cloud across all regions, lasting approximately 10 minutes. The incident impacted only Google Cloud Edge servers; private networking, outbound requests, Metal Hosts, and Dedicated Instances remained unaffected.
Timeline (December 17th, UTC)
- 12:47am: Configuration change merged modifying IP allow-list rules
- 12:49am: Automated alerting notified a Railway engineer
- 12:54am: Change reverted
- 12:58am: Change reapplied
- 12:59am: Automated monitoring recovered
Root Cause
Railway maintained both static and dynamic blocklists over two years. A customer reported intermittent timeouts, traced to their IP being “statically blocked on a list from our ‘pre-dynamic’ era.”
The team attempted removing this legacy list by emptying the array. However, the Google Cloud Terraform Provider treats empty lists as optional parameters that default to applying rules to all traffic. As the article explains, “When a GCP Firewall rule does not specify a source IP range, by default it selects all IPs,” effectively blocking everything instead of an empty set.
Short-term Actions
- Modified paging procedures for faster notification
- Skipped plan phases during Terraform reverts
- Removed the problematic firewall rule and audited remaining legacy rules
Long-term Strategy
Railway is investing in controlled metal infrastructure across multiple datacenters and cloud providers (GCP, AWS trials), reducing dependency on third-party platform configurations.