Incident Report: November 25th, 2025

Author: Brody Over Date: Nov 25, 2025

Impact

The outage affected Railway’s task queue system, causing temporary deployment pauses across Free, Trial, and Hobby tiers. Pro deployments continued but experienced delays. Service-level actions like configuration changes and environment creation were also impacted. All running deployments and platform features remained online.

Incident Timeline

• 22:47 UTC - Engineers alerted as deploy throughput dropped sharply
• 22:50 UTC - Elevated error rates observed across multiple systems
• 23:04 UTC - Free and Trial deployments temporarily disabled
• 23:10 UTC - Hobby deployments paused to reduce system pressure
• 23:16 UTC - Issue identified and mitigation initiated
• 00:06 UTC - Resource reallocation fixes deployed
• 00:21 UTC - Queued deployments began processing
• 00:48 UTC - Queue caught up; all delayed deployments completed
• 00:50 UTC - Hobby deployments restored
• 01:08 UTC - Free and Trial deployments restored
• 01:22 UTC - Incident resolved

What Happened

Around 19:30 UTC, GitHub API latency increased to “nearly 4x their usual p95 latency.” This cascaded into a growing task backlog during peak hours. Workers handling GitHub calls experienced memory failures and crashes. As workers went offline, remaining workers became overloaded, creating a cascade effect.

Railway increased worker CPU and memory resources and adjusted parameters causing workers to request excessive tasks.

Preventative Measures

• Implemented auto-tuning algorithm to prevent worker resource starvation
• Scaled task queue resources across the platform
• Working to remove external API dependencies from critical workflows