Cloudflare incident on June 20, 2024
Authors: Lloyd Wallis, Julien Desgats, Manish Arora
Overview
On June 20, 2024, Cloudflare experienced a significant service disruption lasting 114 minutes. During the 30-minute peak impact window, the company observed that “1.4 - 2.1% of HTTP requests to our CDN received a generic error page, and observed a 3x increase for the 99th percentile Time To First Byte (TTFB) latency.”
Two independent events triggered the outage:
- Automated network monitoring detected performance issues and rerouted traffic suboptimally, causing backbone congestion between 17:33 and 17:50 UTC
- A newly deployed DDoS mitigation rule activated between 14:14 and 17:06 UTC, exposing “a latent bug in our rate limiting system that allowed a specific form of HTTP request to cause a process handling it to enter an infinite loop”
Background
Rate-limiting Suspicious Traffic
Cloudflare applies rate limits to restrict the number of requests visitors can make within specific timeframes. These protections activate through customer configurations or DDoS rule detection. Typically, “rate limits will be applied based on the IP address of the visitor,” though this broad approach can inadvertently block legitimate traffic from shared connections.
Balancing Traffic Across the Network
Cloudflare maintains two primary traffic management systems:
Unimog functions as Cloudflare’s edge load balancer, directing incoming packets to appropriate servers. The system “aims to keep the CPU load uniform across all active servers” within each data center.
Traffic Manager provides global network oversight, considering “overall CPU utilization, HTTP request latency, and bandwidth utilization to instruct rebalancing decisions.” The system includes safety limits to prevent excessive traffic shifts.
Incident Timeline and Impact
All timestamps are UTC on 2024-06-20
- 14:14 — DDoS rule gradual deployment starts
- 17:06 — DDoS rule deployed globally
- 17:47 — First HTTP request handling process is poisoned
- 18:04 — Incident declared automatically based on detected high CPU load
- 18:34 — Service restart shown to recover on a server, full restart tested in one data center
- 18:44 — CPU load normalized in data center after service restart
- 18:51 — Continual global reloads of all servers with many stuck processes begin
- 19:05 — Global eyeball HTTP error rate peaks at 2.1% service unavailable / 3.45% total
- 19:05 — First Traffic Manager actions recovering service
- 19:11 — Global eyeball HTTP error rate halved to 1% service unavailable / 1.96% total
- 19:27 — Global eyeball HTTP error rate reduced to baseline levels
- 19:29 — DDoS rule deployment identified as likely cause of process poisoning
- 19:34 — DDoS rule is fully disabled
- 19:43 — Engineers stop routine restarts of services on servers with many stuck processes
- 20:16 — Incident response stood down
Impact Assessment
The incident affected multiple global data centers. During peak impact, Cloudflare observed error response rates reaching 2.1% and a three-fold increase in 99th percentile latency measurements.
The company documented that “10% of our capacity in Western Europe was already gone, and 4% in Eastern Europe, during peak traffic time for those timezones.”
Technical Description of the Error
The incident originated from a new DDoS mitigation approach combining rate-limits and cookies. The system followed this process:
- Check for valid cookie presence; otherwise block the request
- If valid cookie exists, add a rate-limit rule based on the cookie value
- Apply rate-limit rules after all other DDoS mitigations
The vulnerable code generated keys for rate-limiting lookups. A function called get_cookie_key() contained two critical issues:
Issue One: The DDoS logic’s rate-limit rules used internal APIs “in ways that haven’t been anticipated,” causing the parent key generator to reference the get_cookie_key function itself, creating potential infinite recursion.
Issue Two: The validation function used during key generation differed from earlier validation steps. When clients sent multiple cookies with mixed validity, “both can disagree” on validity status.
When combined, these issues triggered a critical failure. A client sending multiple cookies with some valid and some invalid would cause the broken validation to reject the cookie, triggering the parent key generator call, which recursively called the same function indefinitely.
The request processing logic was written in Lua, a language implementing “proper tail calls,” an optimization where “the final action a function takes is to execute another function.” Rather than adding stack frames, Lua replaces the top frame, meaning “a loop in the request processing logic which never increases the size of the stack” would “simply consumes 100% of available CPU resources, and never terminates.”
Once a process received a request matching these conditions, it became “poisoned” and could not process further requests. While individual processes had minimal impact, the cascading effect proved severe:
- Unimog reduced traffic to affected servers, redistributing load elsewhere
- Traffic Manager redirected traffic to other data centers
- Redirected traffic included the poisoning requests, spreading the failure globally
- Within minutes, multiple data centers experienced widespread process poisoning
Resolution and Response
Engineers were automatically notified at 18:04 UTC when global CPU utilization reached sustained elevated levels. Initial investigation considered correlation with the earlier network congestion event. The breakthrough came when investigators recognized that locations with highest CPU showed lowest traffic — contradicting a network-based cause.
The response strategy included:
- Testing whether process restarts resolved poisoning (confirmed effective after 25 minutes)
- Initiating mass rolling restarts of affected services
- Identifying the DDoS rule deployment as the trigger
At 19:34 UTC, the new DDoS rule was fully disabled. Additionally, “conditions presented by the incident triggered a latent bug in Traffic Manager,” causing the system to halt routing actions during two windows (18:35-18:52 UTC and 18:56-19:05 UTC). The Traffic team deployed a fix by 19:05 UTC.
Additional Technical Details
During the incident, Traffic Manager data revealed the geographic scope of the problem. A capacity actions map from 18:09 UTC showed significant traffic rerouting patterns, with orange ribbons indicating traffic flow between data centers and circle colors representing CPU load status.
The article notes that “partially poisoned servers in many locations struggled with the request load, and the remaining processes could not keep up, resulting in Cloudflare returning minimal HTTP error responses.”
Remediation and Follow-up Steps
Cloudflare implemented immediate and long-term improvements:
Immediate Actions:
- Fully rolled back the new DDoS rule
- Committed to disabling reactivation until the broken cookie validation is fixed
Design Improvements:
- Recognized that “the rate limiting implementation in use for our DDoS module is a legacy component” compared to newer customer-facing engines
- Exploring options to “limit the ability to loop forever through tail calls”
- Commencing “early implementation stages of replacing this service entirely,” with new designs that “will allow us to apply limits on the non-interrupted and total execution time of a single request”
Process Enhancements:
- Noted that the new rule was “staged in a handful of production data centers for validation, and then to all data centers a few hours later”
- Committed to enhanced staging and rollout procedures “to minimize the potential change-related blast radius”
Conclusion
Cloudflare stated: “We are sorry for any disruption this caused our customers and to end users trying to access services.” The company confirmed that “the conditions necessary to activate the latent bug in the faulty service are no longer possible in our production environment, and we are putting further fixes and detections in place as soon as possible.”