Incident Report: October 16th, 2025
Author: Chandrika Khanduri Date: Oct 16, 2025
Impact
Railway’s Edge Network experienced connectivity disruptions affecting all regions. Users attempting to access Railway-hosted domains via HTTP and TCP “may have experienced intermittent connection failures or timeouts for approximately 2-3 minutes.” Private networking remained unaffected, though some users received HTTP 404 errors, HTTP 522 errors, or Cloudflare error pages during the outage window.
Incident Timeline
- 20:50 UTC – Spike detected in external traffic across the proxy fleet
- 20:51 UTC – Internal monitoring probes failed; engineers alerted
- 20:51 UTC – Routing services reached high memory utilization; outage commenced
- 20:52 UTC – Load balancers halted traffic forwarding due to failed health checks
- 20:53 UTC – Service recovered; full restoration achieved across regions
Root Cause
The incident stemmed from “increased load on our edge network that exhausted available memory in a supporting service,” coinciding with a routine edge network upgrade. When a traffic spike occurred, the already-strained routing service couldn’t initialize new proxies properly, triggering health check failures and load balancer traffic cutoffs.
Preventative Measures
Immediate Actions:
- Increased internal routing service memory limits
- Enhanced memory utilization monitoring with escalating alerts
- Added safeguards preventing routing service overload
Q4 2025 Roadmap: The team is completely rewriting the routing service for efficiency and implementing multi-region distributed architecture to prevent platform-wide failures.