Incident Report: July 2, 2026 — US East Services Outage

Author: Ray Chen Date: Jul 3, 2026

Impact

Between approximately 07:44 UTC and 12:01 UTC on July 2, 2026, Railway users experienced increased response times and intermittent connectivity issues affecting traffic between US regions, including private networking. One US East availability zone additionally suffered degraded disk performance and disrupted private networking for approximately two hours.

Incident Timeline

All times are UTC on July 2, 2026:

• 07:44 — Packet loss detected in US East region; public incident declared. Issue traced to upstream network carrier.
• 07:44–08:32 — Disconnection from degraded carrier at all US border routers; traffic rerouted through other carriers. Most US paths improved, though latency and packet loss into US East persisted.
• 08:39 — Secondary carrier at affected US East zone disconnected due to continued packet loss. Unintentionally removed the zone’s last default route to the internet.
• 08:59 — Secondary carrier reconnected; routing stabilized and connectivity began recovering.
• 09:00–10:45 — Storage performance remained degraded despite healthy routing; throughput capped at approximately one-third capacity.
• 10:45 — Root cause identified: storage connections established over slow management network during instability remained stuck there.
• 10:45–11:00 — Stuck management-network connections terminated across storage and compute hosts; reconnection over correct network occurred within seconds.
• 11:04 — I/O wait returned to baseline; storage throughput normalized. Private networking tunnel address issue identified.
• 11:49 — Mesh networking agents restarted fleet-wide in zone; private networking fully recovered.
• 12:01 — Incident marked resolved with all metrics stable.

What Happened

Three distinct failure mechanisms contributed to cascading effects:

1) Upstream ISP Degradation

Railway connects every datacenter to multiple Tier 1 ISPs for redundancy. On July 2, one carrier experienced network degradation in their US backbone, causing traffic spillover that saturated the route carrying Railway’s US West-East traffic. This created elevated latency and packet loss. After detection and disconnection at all US borders, traffic successfully rerouted through alternative providers.

2) Storage Performance Degradation

At 08:39, disconnection from a secondary carrier at the affected zone inadvertently removed the site’s last default route to the internet. The affected zone, being a first-generation datacenter, depends on carriers rather than self-generating default routes—unlike newer facilities. For approximately 20 minutes, the site lacked stable internet connectivity.

During this instability, a hidden bug emerged: when the primary network fabric’s default route disappeared, servers fell back to their management network. Linux’s default behavior allowed servers to answer for storage addresses on that slower network, causing storage traffic to flow over a path with a fraction of the fabric’s capacity. Network connections don’t re-check routes once established, so connections created during this window remained stuck on the management network even after full fabric restoration.

Storage throughput remained capped at approximately one-third of normal capacity until stuck connections were identified and terminated, forcing reconnection over the correct fabric network. I/O wait dropped from 58% to under 5% within approximately 15 minutes.

3) Private Networking Degradation

Railway’s private networking operates via encrypted tunnels between servers, with peer addresses learned from received packets. During routing disturbance, thousands of tunnels learned incorrect peer addresses from a device rewriting source addresses. These broken tunnels persisted because the mesh only re-verifies addresses during membership changes, and idle tunnels never trigger correction. Approximately 20,000 host-to-host private network links were affected at peak, requiring mesh networking agent restarts to re-establish correct tunnel addresses.

Preventative Measures

Already implemented:

• Disconnected degraded carrier at all US borders; fleet currently operates on other Tier 1 carriers with full capacity headroom
• Cleared stuck management-network storage connections across affected zone
• Restarted mesh networking agents fleet-wide in the zone

In progress:

• Migrating first-generation sites to self-generated default routes, eliminating carrier dependency
• Correcting host fallback behavior to prevent production traffic from using the management network as fallback
• Adding alerting on management network utilization and blackholed private network links for immediate detection

Conclusion

The incident originated from an upstream carrier failure—a normal internet event. However, cascading effects resulted from site infrastructure decisions (carrier-dependent default routes), operational choices (disconnection without verifying remaining routes), and system designs that silently captured and retained incorrect paths during instability rather than re-asserting correct state after recovery.