Incident Report: October 15th, 2025

Author: Brody Over Date: Oct 15, 2025

Impact

From 20:49 to 22:15 UTC, Railway users experienced an outage affecting dashboard access and deployment capabilities. According to the report, “running services and networking remained operational” despite most control systems being unavailable during this window.

Incident Timeline

The incident began when “a large deletion operation triggered a surge in internal requests, causing elevated error rates.” The on-call team was alerted immediately and identified the root cause as unexpected overload on network management systems by 21:06 UTC.

Key recovery milestones included:

• 21:08 UTC: Deployments paused to reduce pressure
• 21:45 UTC: Fix implemented and deployed
• 22:00 UTC: Platform began recovery with gradual deployment re-enablement
• 22:15 UTC: Majority of users regained full access
• 23:05 UTC: Residual cache issues resolved

Preventative Measures

Railway committed to implementing “improved rate limiting of internal APIs” and “better visibility and alerting around internal request patterns” to prevent recurrence. The company also pledged to make the network control plane more resilient overall.

The statement concluded: “We apologize for any inconvenience caused” and emphasized commitment to eliminating this class of issues.