Incident Report: December 8th, 2025

Authors: Noah Dunnagan & Ray Chen Date: Dec 8, 2025

Impact

The outage disrupted Railway’s backend API, affecting “dashboard access, CLI operations, GitHub-based deployment processing, login, and API functionality.” Notably, running deployments remained operational, and users avoiding dashboard interactions experienced no problems.

Incident Timeline

• 15:03 UTC – Engineers merged a database schema change with data migration
• 15:17 UTC – Migration began processing
• 15:18 UTC – “Lock conflicts in the database caused backend replicas to become unhealthy”
• 15:26 UTC – Migration completed; replicas recovered

Root Cause

Railway deployed a migration adding a nullable column to a heavily-used table containing approximately 1 billion rows. A long-running query held locks, blocking the migration. Connection attempts accumulated until “PgBouncer exceeded the database’s connection limit,” causing replicas to fail health checks and be removed from traffic rotation.

Remediation

Once the prolonged query released its locks, the migration applied immediately, allowing replicas to reconnect and resume serving traffic.

Preventative Measures

• Increase database connections and reconfigure PgBouncer
• Fix the long-running query to prevent future row lock conflicts