Incident Report: August 27th, 2024

Date: August 27, 2024 Author: Jacob Cooper

Summary

Railway experienced an outage affecting approximately 30% of traffic for roughly 30 minutes, with full recovery taking about 60 minutes. The incident primarily impacted new customers using the new proxy or public TCP proxy services.

Timeline of Events

• 10:04 PM UTC: Engineer merged a pull request to add more proxies
• 10:08 PM UTC: On-call engineer paged for synthetic testing failures related to TCP proxy probes
• 10:13 PM UTC: Root cause identified—the PR had recreated all new Railway proxy instances simultaneously
• 10:44 PM UTC: Instances re-bootstrapped; partial customer recovery began
• 10:49 PM UTC: Status page updated after sufficient proxy recovery in each region
• 11:17 PM UTC: US West fully recovered
• 11:32 PM UTC: Singapore fully recovered
• 11:34 PM UTC: Europe and US-East fully recovered
• 11:40 PM UTC: Incident declared resolved

Root Cause

A pull request modifying proxy instance counts triggered infrastructure recreation. An earlier, separate PR that had resized boot disks on Google’s dashboard contained conflicting configurations. When merged, the external IaC vendor used outdated boot disk specifications, causing machines to recreate “with live traffic.”

Failed Safeguards

Incorrectly Logged Plan: The IaC platform reported “4 to add, 29 to change, 0 to destroy,” but the actual Terraform plan showed “4 to add, 12 to change, 17 to destroy.” This discrepancy went undetected.

Missing Deletion Protection: Deletion protection was only enabled in production environments, not staging. The newproxy resource lacked this configuration, allowing destructive changes to proceed unchecked.

Remediation Actions

• Implement deletion protection across all Terraform policies with production resources
• Remove destroy capabilities from internal Terraform deployment boxes, requiring privileged escalation for resource deletion
• Add supplemental alerting for these failures
• Create machine images for new proxies enabling faster deployment versus just-in-time bootstrapping