Railway Incident Report: April 23rd, 2025

Date: April 23, 2025 Author: Ray Chen

Overview

Railway experienced a major outage affecting its networking control plane, making “public and private networking on Railway was unavailable for a portion of users.”

Impact

The outage impacted Edge Network, Private Network, and TCP Proxy across all regions. The Asia region experienced the highest impact, while other regions saw intermittent unavailability for a small subset of users. Additionally, users couldn’t provision new domains or SSL certificates during this period.

Timeline

• 09:07 UTC: Google Cloud SQL initiated maintenance on the Postgres service
• 09:08 UTC: Database-related errors appeared across control plane instances
• 09:10 UTC: On-call engineers paged for network monitoring failures
• 09:11 UTC: Private Network monitor failures detected
• 09:15 UTC: Partial recovery observed
• 09:17 UTC: Maintenance completed (10-minute window)
• 09:28 UTC: Full recovery across platform
• 10:36 UTC: Incident fully resolved

Root Cause

The control plane’s Postgres database became unreachable during Google Cloud SQL maintenance that affected the read replica. Some control plane instances crashed due to improper error handling, evicting routing information from memory. Since the database was offline, instances couldn’t reload this critical data.

Preventative Measures

Railway outlined plans to restructure their control plane architecture for 2025 Q2, including:

• Creating regional control planes instead of a global one
• Fixing database connection error handling
• Adding persistent cache layers for routing information
• Reducing control plane startup time
• Coordinating with Google Cloud SQL to prevent simultaneous maintenance windows