Incident Report: January 31st, 2024

Date: Jan 31, 2024 Author: Angelo Saraceno

Summary

Railway experienced two significant DDoS attacks between January 28-31, 2024. The more severe incident on January 31st involved “a L4 (TCP) SYN flood attack” that peaked at approximately 12 million requests per second, affecting European users from 18:13 UTC to 23:46 UTC.

Key Impact

The primary outage prevented “access to workloads hosted on Railway if a connection required new workloads” during the specified timeframe. European users experienced delayed or dropped requests, though global traffic from other regions continued functioning.

Root Cause

The attack utilized what Railway believed to be a Mirai botnet, characterized by massive SYN packets that exhausted server resources. The proxy servers accumulated approximately 130,000 connections in a SYN state, overwhelming the European shard of their edge network.

Resolution Timeline

• 18:13 UTC: Alert triggered; investigation began
• 18:59 UTC: Initial mitigation attempt using previous solutions
• 19:23 UTC: Attack source identified as EU edge proxy hosts
• 20:21 UTC: Additional services inserted between serving layers
• 21:32 UTC: Migration strategy revised for gradual restoration
• 23:46 UTC: Full service restoration confirmed

Remediation Steps

Engineers modified kernel TCP settings, implemented aggressive rate limiting, and inserted an additional service layer to filter abusive traffic before reaching user workloads.