Incident Report: May 19, 2026 - GCP Account Suspension

Authors: Chandrika Khanduri & Cody De Arkland Date: May 20, 2026

Impact

Between 22:20 UTC on May 19 and 06:14 UTC on May 20 (approximately 8 hours), Railway experienced a platform-wide outage after Google Cloud suspended the company’s production account. The suspension rendered the API, control plane, and databases offline, along with compute infrastructure hosted on GCP.

Users encountered 503 errors and were unable to log in. Although workloads on Railway Metal and AWS infrastructure remained operational, the cascading effect was severe: “all Railway workloads across all regions were rendered unreachable” when cached routing tables expired.

Incident Timeline

• 22:10 UTC: Automated monitoring detected API health check failures
• 22:11 UTC: Dashboard returned 503 errors; users unable to access accounts
• 22:19 UTC: Root cause identified—Google Cloud suspended Railway’s production account
• 22:22 UTC: P0 ticket filed with Google Cloud
• 22:29 UTC: Incident declared; GCP account access restored (but services remained offline)
• 23:54 UTC: All persistent disks restored to ready state
• 01:30 UTC (May 20): Compute instances began recovering
• 01:38 UTC: Edge traffic restoration and networking recovery
• 02:47 UTC: GitHub rate-limiting triggered, blocking logins and builds
• 02:55 UTC: Dashboard accessible again
• 04:00 UTC: API, dashboard, and OAuth endpoints confirmed operational
• 07:58 UTC: Incident fully resolved

Root Cause

Google Cloud “placed Railway’s production account into a suspended status incorrectly, as part of an automated action” affecting multiple customers platform-wide without advance notice.

The deeper architectural vulnerability: Railway’s network control plane, hosted on GCP, served as a single point of failure. Despite maintaining a mesh ring topology connecting Metal, GCP, and AWS infrastructure, workload discoverability depended entirely on the GCP-hosted control plane API. When cache expired after one hour, the mesh could no longer populate routing tables.

Preventative Measures

Railway committed to three architectural changes:

1. Remove GCP dependency from the mesh: Make the network topology a true mesh where any interconnect failure doesn’t prevent route re-population

2. Distribute database infrastructure: Extend high-availability database shards across AWS and Metal, ensuring quorum operations survive complete cloud provider failures

3. Restructure the data and control planes: Remove Google Cloud services from the critical path for core user-facing components, relegating them to secondary/failover roles only

The company stated: “Your uptime is our responsibility, and we’ll keep delivering on it.”