Incident Report: November 20th, 2025
Author: Noah Dunnagan Date: Nov 21, 2025
Impact
Railway’s deployment system experienced temporary disruptions on November 20th, 2025. The company reported that “all deployments on Railway were temporarily delayed during the outage due to an issue with our deployment task queue.” However, running services remained operational. Users who didn’t push code or trigger redeployments faced no service interruption.
Timeline
The incident unfolded across several hours:
- 16:54 UTC - Engineers detected unusually low GitHub webhook activity; deployments became stuck
- 17:29 UTC - Webhook traffic from GitHub spiked 10x, causing deployment initialization overwhelm
- 17:32 UTC - Team escalated due to growing deployment backlog
- 17:41 UTC - Free, Trial, and Hobby deployments disabled to reduce queue pressure
- 18:19 UTC - Pro deployments temporarily halted
- 18:47 UTC - Pro deployments restored
- 19:01 UTC - Hobby deployments restored
- 19:05 UTC - Free and Trial deployments restored
- 19:18 UTC - Full recovery confirmed
Root Cause
Railway processes deployments through an asynchronous task queue. After an initial dip in GitHub webhook delivery, a surge of events arrived simultaneously, “triggering massive deployment creation, overwhelming our deployment processing pipeline.” Workers experienced memory pressure and lockups during this flood.
Response & Prevention
The team implemented graceful service degradation, prioritizing Enterprise and Pro users. Going forward, Railway plans to: enhance deployment queue alerts, add monitoring for unusual spikes, and fix worker memory pressure issues.