Incident Report: October 28th, 2025

Author: Ray Chen Date: Oct 28, 2025

Impact

The outage affected multiple Railway systems: “the dashboard inaccessible, caused CLI operations relying on our API to fail (e.g. railway up), delayed GitHub-based deployments due to webhook processing failures, and disrupted all Railway Public API operations.”

However, running deployments remained online, and users who avoided the dashboard and CLI experienced no disruption.

Incident Timeline

• 18:23 UTC - Database schema modification introduced
• 18:34 UTC - Change deployed to production
• 18:36 UTC - Monitoring alerts triggered
• 18:41 UTC - Incident declared
• 18:56 UTC - Root cause identified
• 19:00 UTC - Service recovery began
• 19:15 UTC - Full restoration confirmed

Root Cause

A database migration added “a new column with an index to a table containing approximately 1 billion records” without using Postgres’ CONCURRENTLY option. This created an exclusive table lock, queuing all queries. The system exhausted connection slots, preventing administrative access to terminate the problematic operation. After 30 minutes, the migration completed naturally, releasing the lock.

Preventative Measures

Railway committed to three changes:

1. Enforcing CONCURRENTLY in CI for index operations
2. Adjusting PgBouncer connection limits
3. Configuring reserved administrative connections for incident response