Cloudflare Incident on November 14, 2024, Resulting in Lost Logs

Authors: Jamie Herre, Tom Walwyn, Christian Endres, Gabriele Viglianisi, Mik Kocikowski, Rian van der Merwe

Overview

On November 14, 2024, Cloudflare experienced a significant outage affecting Cloudflare Logs services. During approximately 3.5 hours of impact, “about 55% of the logs we normally send to customers were not sent and were lost.” The incident demonstrates how misconfiguration in one system can cascade through interconnected infrastructure when safeguards are not properly activated.

Background

Cloudflare’s globally distributed network generates extensive event logs across its infrastructure. “On a typical day, Cloudflare sends about 4.5 trillion individual event logs to customers.” These logs serve critical functions including compliance verification, system observability, and accounting purposes.

The company offers two primary log delivery mechanisms:

• Edge Log Delivery: Sends logs directly from individual servers
• Logpush: Collects and pushes logs in predictable file sizes with automatic scaling

System Architecture

The logging pipeline comprises four interconnected services:

Logfwdr

A Golang service that accepts event logs from Cloudflare’s global network and forwards them in batches. It determines which logs should be forwarded based on event type, customer identity, and routing rules, relying on configuration data for decision-making.

Logreceiver

Also written in Golang, this service accepts log batches from across the network and sorts them by event type and purpose. For Cloudflare Logs, it demultiplexes batches into per-customer collections and forwards them to Buftee. “Currently, Logreceiver is handling about 45 PB (uncompressed) of customer event logs each day.”

Buftee

An internal buffering system serving multiple purposes in the data pipeline. “Buftee provides buffers for each Logpush job, containing 100% of the logs generated by the zone or account referenced by each job.” This design prevents one customer’s processing failures from affecting others and avoids head-of-line blocking. The service typically manages over 1 million buffers globally.

Logpush

A Golang service that reads logs from Buftee buffers and pushes results in batches to customer-configured destinations. “Currently, we push over 600 million such batches each day.”

What Happened

The incident began when Cloudflare added support for a new dataset in Logpush, requiring additional configuration for Logfwdr. A configuration generation system produced a blank configuration file, essentially signaling that no customers had logs configured for pushing. The team identified and reverted the change within five minutes.

However, this brief misconfiguration triggered a latent failsafe mechanism in Logfwdr. The failsafe was “designed to protect against a situation when this specific Logfwdr configuration was unavailable… by transmitting events for all customers instead of just those who had configured a Logpush job.”

This failsafe, implemented during earlier, lower-traffic periods, caused Logfwdr to send logs for every customer simultaneously. The resulting massive spike created approximately 40 times more buffers than normal — roughly 40 million buffers globally compared to the typical baseline.

Buftee, designed to prevent cascading failures, lacked proper configuration to handle this buffer explosion. The system became overwhelmed and unresponsive, requiring a complete reset and restart to recover. The entire process took several hours despite the initial five-minute configuration error.

Root Causes

The analysis identified two primary failures:

First Failure: A bug in the configuration generation system created the blank configuration. While this type of bug was anticipated and addressed through failsafe design, the broader system was never tested under fail-open conditions.

Second Failure: Buftee possessed protective mechanisms against buffer explosion but they were not activated. The company acknowledges this represented a critical gap: “while we had the safeguard of Buftee in place, we hadn’t ‘buckled it up’ by configuring the necessary settings.”

The incident illustrates a classic systems design problem: protective mechanisms are only effective when properly configured and maintained.

Going Forward

Cloudflare is implementing multiple preventive measures:

1. Monitoring and Alerts: Creating alerts to catch specific misconfigurations before they propagate through the system.

2. Bug Fixes and Testing: Addressing the underlying bug and expanding associated test coverage.

3. Resilience Testing: Expanding beyond current disaster recovery practices. “Currently, we conduct regular ‘cut tests’ to ensure that these systems will cope with the loss of a datacenter or a network failure. In the future, we’ll also conduct regular ‘overload tests’ to simulate the kind of cascade which happened in this incident.”

The company emphasizes that accepting inevitable human error requires systems designed to respond “predictably and gracefully” to misconfigurations and failures.