Incident Report: May 4th, 2024

Published: May 8, 2024

Overview

Railway experienced a regional outage affecting the Asia-Southeast cluster. The incident impacted “1% of our compute nodes and less than 1% of our workloads,” though the region itself experienced more significant disruption requiring public disclosure per company policy.

Timeline

The incident unfolded across approximately 7.5 hours:

• 03:39 UTC: On-call engineer alerted to exhausted compute capacity and low-disk alarms in Asia-Southeast
• 04:13 UTC: Partial outage declared; all instances showed unexpectedly high memory usage (above 95%)
• 04:30 UTC: New compute node deployed; high-memory workloads manually removed
• 05:06 UTC: Mitigation efforts communicated
• 05:58 UTC: Partial recovery achieved; one compute node remained unresponsive
• 06:05 UTC: Service recovery began
• 08:10 UTC: 50% capacity restored; investigation continued on remaining issues
• 10:08 UTC: Problematic user workload identified
• 10:12 UTC: More aggressive resource limits applied; node recovery successful
• 10:25 UTC: Builder nodes cycled; deployments restored
• 10:32 UTC: Configuration issue fixed
• 10:57 UTC: Incident resolved

Root Cause

A single user workload triggered a cascade of failures:

1. At 03:30 UTC, the workload deployed multiple replicas across all compute nodes, consuming full RAM allocations
2. This caused kernel memory compaction processes to consume excessive CPU
3. The kernel OOM reaper killed critical daemon processes, making all compute nodes unhealthy
4. A subsequent deployment at 05:54 UTC concentrated the workload on a single node, causing memory pressure that locked the hypervisor

The core problem: “the total amount of container resource limits we allocated was above the physical RAM allocated to the compute node,” so no per-container limits were breached despite system memory reaching 99% utilization.

Remediation

Railway implemented several preventative measures:

• Adjusted kernel OOM reaper configuration to prioritize user workload eviction over system processes
• Planned system memory pressure thresholds for workload migration
• Designed deployment algorithms incorporating historical metrics
• Increased Asia-Southeast compute capacity for greater burst tolerance
• Enhanced alerting for smaller regions

Communications

The team issued six status updates via status.railway.app but acknowledged this pace was insufficient. Moving forward, they commit to “a policy and automated ChatOps prompt to update affected customers every 30 minutes at a minimum.”