Incident Report: June 11th, 2024
Author: Jacob Cooper Date: Jun 11, 2024
Overview
Railway experienced a significant platform outage affecting US and EU infrastructure. Approximately 20% of instances became degraded with slow request handling. In rare cases, some machines required full restarts, causing complete application downtime.
Incident Response Timeline
The incident began at 1:41 AM UTC when an engineer was alerted to IO pressure on a European instance. A service consuming approximately 1TB of ephemeral storage was identified and redeployed by 2:31 AM UTC.
A second wave emerged at 7:53 AM UTC with elevated IO pressure across multiple US-west instances. An incident was formally declared at 8:24 AM UTC after identifying an unresponsive machine. “Some workloads saw up to 20m downtime” during recovery efforts.
Key timeline milestones:
- 9:03 AM UTC: Hobby provisioning disabled to prioritize Pro plan workloads
- 10:45 AM UTC: Last machine restarted
- 11:57 AM UTC: Instances confirmed operational; private networking issue identified affecting ~5% of machines
- 12:29 PM UTC: Private networking restored
- 1:00 PM UTC: Incident declared resolved
Root Cause
The outage stemmed from “an errant migration workflow, which exacerbated IO pressure” by triggering excessive concurrent operations. Multiple contributing factors converged simultaneously:
- Standard deployments filled the deployment queue
- Scheduled crons activated hourly
- European placement failures triggered secondary queue backups
- Host migration scripts executed concurrently with above processes
The migration workflow contained critical bugs: it triggered redeploys per replica rather than per deployment, causing exponential queue growth. The status endpoint—already IO-intensive—was hammered with 4x normal load, rendering parts of the platform unschedulable and cascading failures across regions.
Solutions Implemented
Railway’s infrastructure team deployed several fixes:
- Cache machine status responses to reduce IO operations
- Implement global rate limits on worker processes
- Configure scheduler to query only same-region workloads
- Optimize resource eviction prioritizing stateless Hobby workloads
The company engaged directly with affected customers and invited feedback on incident response procedures.