Post Mortem: What Yesterday’s Network Outage Looked Like

Author: Matthew Prince

Overview

Cloudflare experienced a network service interruption around 16:36 GMT caused by upstream provider issues affecting European data centers combined with a misapplied rate limit during DDoS mitigation efforts. The company acknowledged the incident fell short of their reliability standards and outlined steps taken to prevent recurrence.

Two Visible Events

Traffic graphs across Cloudflare’s eight European data centers revealed two distinct anomalies: a significant inbound traffic spike around 13:30 GMT and outbound traffic decline near 16:30 GMT. While separated by three hours, these events were interconnected parts of the same incident sequence.

Limited Network and a Nasty Attack

An upstream network provider experienced issues in Europe, forcing traffic concentration in regional facilities. Around 13:00 GMT, a major DDoS attack targeting a customer website began as a “layer 4 attack” exceeding 65 Gbps, primarily concentrated in Europe. Cloudflare’s systems successfully mitigated this initial vector without impacting other customers.

Mitigation and a Mistake

As the attacker shifted strategies, Cloudflare’s ops team manually adjusted routing rules. Around 16:30 GMT, they implemented a new rate limit intended for the targeted customer but “was misapplied to a wider number of customers,” causing widespread European service degradation with spillover to other regions.

Smoke Tests

The incident revealed a gap in automated safeguards. Cloudflare committed to implementing additional “smoke tests” to catch similar deployment errors before reaching production systems.