Incident Report: Dec 1st, 2023

Author: Angelo Saraceno Date: Dec 1, 2023

Summary

Between 16:40 UTC and 20:53 UTC, Railway experienced an outage affecting 30% of US-West compute infrastructure. The company stated: “users were unable to connect to their workloads” due to “high resource usage leading to a soft lock of all cores on the affected hosts.”

Customer workloads experienced downtime “ranging from 10 minutes to 1 hour” during mitigation efforts. The root cause was traced to “a newly deployed metrics collection agent that appeared to trigger a CPU core soft lock on these hosts, despite them being well below resource thresholds,” with Railway believing the underlying issue involved the Google Cloud Hypervisor.

Incident Details

On December 1st at 16:02 UTC, Railway began rolling out a new metrics collection service. During deployment, hosts became unavailable. The problem emerged when “a configuration error meant it began analyzing processes beyond its original mandate — essentially sampling all processes on a host at a high frequency.”

This triggered “cascading CPU core soft locks which brought the host down.” The issue affected only a random subset of machines (30% of the fleet), making diagnosis difficult since the metrics collector functioned properly on 70% of infrastructure.

Response and Resolution

Railway’s engineering team initiated manual host restarts at 17:05 UTC, taking “between 10-30 minutes per host” to recover systems. By 20:23 UTC all hosts were restored with no data loss reported. The incident was declared resolved at 20:53 UTC.

Preventative Measures

Railway identified three key improvements:

• Enhance internal deployment tooling to identify issues faster
• Build better mechanisms for detecting and automatically recovering from infrastructure failures
• Develop cloud-independent systems and bare metal capabilities for greater stack visibility

Incoming Mitigations

The company will implement staged rollouts across all services, beginning at 10%, progressing to 25%, 50%, and finally 100%, with careful monitoring at each stage.