Incident Report: June 6th, 2025
Author: Jacob Cooper Date: Jun 6, 2025
Impact
The outage affected dashboard access and login functionality. While deployments and webhook-based builds continued operating normally, users experienced slow requests and timeouts when accessing the dashboard. Upon re-authentication attempts via GitHub, sessions were cleared and users were fully logged out. During the incident, “GitHub rate limits were in affect, preventing new logins (while previously logged in sessions remained valid).”
Incident Timeline
- 4:31pm UTC: Database connection exhaustion notification received
- 4:34pm UTC: High CPU usage identified in pooling layer with 5-minute periodic pattern
- 5:08pm UTC: Incident declared after GitHub login failures
- 6:49pm UTC: Logins restored
- 6:58pm UTC: Incident resolved
- 7:41pm UTC: Incident reopened due to additional failed login reports
- 9:00pm UTC: Partial recovery initiated
- 9:16pm UTC: Full recovery completed
- 9:57pm UTC: Incident closed following extended monitoring
What Happened
Rapid scaling over six months created database pressure. When “circuit breakers triggered, aggressive websocket reconnect logic initiated on every connected client at once.” Websocket handshakes weren’t covered by WAF policies, allowing “hundreds of thousands of users bypassed our firewall rules, resulting in millions of requests to our backend.”
The cascade effect: timeouts triggered login page serving, which sent users to GitHub’s OAuth endpoints, causing secondary rate limiting on Railway’s app. The company faced a dilemma—restoring the backend risked renewed request floods from outdated clients, while keeping circuit breakers active perpetuated the GitHub rate-limiting loop.
Preventative Measures
Already implemented:
- GitHub login rate limiting queues
- Websocket reconnect logic backoff adjustments
- Modified rate limiting rules
- Read replica utilization for real-time logic
Planned (within 24 hours):
- WAF configuration failure detection testing
- Login buttons served only when backend is operational