Incident Report: April 30th, 2025

Author: Ray Chen Date: Apr 30, 2025

Impact

Railway’s API backend experienced an outage affecting the dashboard, CLI, and Public GraphQL API. Users encountered “connection errors if you were using the Railway dashboard, the CLI, or Railway’s Public GraphQL API during this period.” GitHub-based deployments were temporarily delayed, though running deployments and platform networking remained operational.

Incident Timeline

• 19:50 UTC: Dashboard loading failures detected; health check monitors failed
• 20:00 UTC: Incident declared as Partially Degraded
• 20:36 UTC: Configuration rollbacks initiated
• 21:48 UTC: Incident upgraded to Major Outage
• 22:20 UTC: Initial recovery appeared successful
• 22:24 UTC: Health check monitors failed again
• 22:49-23:23 UTC: Additional configuration rollbacks performed
• 23:40 UTC: Incident fully resolved

Root Cause

Recent configuration changes to the Kubernetes-based API backend created database connection failures. “Our Kubernetes pods were repeatedly crashing with database connection errors some time after they successfully start up.” The service mesh failed to properly inject DNS routing information into pgbouncer pods, causing connection pooler failures. Modified healthchecks incorrectly marked unhealthy instances as healthy, overloading the Postgres database.

Remediation

Railway fully reverted configuration changes and cycled all pods to resolve service mesh DNS propagation issues. The service mesh was subsequently removed from the API backend.

Preventative Measures

• Staggered rollout of configuration changes based on risk level
• Mandatory rollback plans for each configuration change
• Immediate full reversion rather than selective rollbacks
• Complete removal of the service mesh from critical paths