Incident Report: September 22nd, 2025
Author: Ray Chen | Date: Sep 22, 2025
Impact
A significant outage affected Railway’s dashboard and deployment pipeline. Users encountered infinite loading states when accessing project canvases, delayed deployments, and “Limited Access” messages when initiating new deployments. However, “all running deployments and platform-level networking features remained online throughout this period.”
Incident Timeline
The incident unfolded across multiple phases:
- 11:05 UTC – Initial reports of project canvas failures
- 11:14 UTC – Partial outage declared; investigation began
- 11:35 UTC – Root cause identified and remediated
- 11:45 UTC – Full service restoration achieved
- 11:48 UTC – New Control Plane issues emerged with abnormal resource consumption
- 12:14 UTC – Dashboard and Control Plane issues correlated
- 12:25 UTC – Degradation intensified; all deployments suspended
- 12:44 UTC – Pro tier deployments restored
- 12:50 UTC – Non-Pro tier deployments restored
- 13:05 UTC – Non-Pro deployments rolled back due to persistent instability
- 14:01 UTC – Complete recovery achieved
- 21:27 UTC – Private network visibility issue fixed
Root Cause Analysis
The incident began with “a database schema change to remove an unused column” during code cleanup. A subsequent code change unintentionally reverted this modification, causing failures in a critical API supporting the frontend.
This cascaded globally, triggering deployment failures across all interfaces. When the initial fix deployed, a substantial backlog of queued deployments triggered simultaneously, overwhelming the Control Plane’s resources.
Further investigation identified a problematic PgBouncer version upgrade. After migrating from deprecated Bitnami images to an internal repository, the system advanced “several PgBouncer versions ahead.” The upgraded version exhibited “abnormally high CPU and memory consumption, manifesting as increased latency and database connection timeouts.”
Response and Remediation
Railway implemented a phased approach: temporarily suspending non-Pro deployments, then all deployments, while scaling Control Plane database resources. The team ultimately rolled back to the previous stable PgBouncer version, achieving “a 100x performance increase with all metrics returning to nominal values.”
Preventative Measures
Future protections include enhanced CI pipeline validation for schema changes, monitoring for PgBouncer performance metrics, and phased rollout strategies for version upgrades. Additionally, Railway is “re-engineer[ing] parts of our Control Plane to be more resilient,” including moving storage away from Postgres.