A deep dive into Cloudflare’s September 12, 2025 dashboard and API outage
Authors: Tom Lianza and Joaquin Madruga
What Happened
Cloudflare experienced an outage affecting the Tenant Service API, which cascaded to impact many other APIs and the Dashboard itself.
The root cause involved a bug in the dashboard code. According to the post, “This bug caused repeated, unnecessary calls to the Tenant Service API” due to a React useEffect hook that included “a problematic object in its dependency array.” Because this object was recreated constantly, it triggered excessive API calls during each render cycle rather than executing once as intended.
The timing proved critical: this dashboard bug coincided with a Tenant Service API update, creating compounding instability that eventually overwhelmed the service. Since the Tenant Service handles API request authorization, its failure cascaded broadly — when it couldn’t evaluate authorization, API requests returned 5xx status codes.
Timeline
| Time (UTC) | Description |
|---|---|
| 2025-09-12 16:32 | A new version of the Cloudflare Dashboard is released which contains a bug that will trigger many more calls to the /organizations endpoint, including retries in the event of failure. |
| 2025-09-12 17:50 | A new version of the Tenant API Service is deployed. |
| 2025-09-12 17:57 | The Tenant API Service becomes overwhelmed as new versions are deploying. Dashboard Availability begins to drop IMPACT START |
| 2025-09-12 18:17 | After providing more resources to the Tenant API Service, the Cloudflare API climbs to 98% availability, but the dashboard does not recover. IMPACT DECREASE |
| 2025-09-12 18:58 | In an attempt to restore dashboard availability, some erroring codepaths were removed and a new version of the Tenant Service is released. This was ultimately a bad change and causes API Impact again. IMPACT INCREASE |
| 2025-09-12 19:01 | In an effort to relieve traffic against the Tenant API Service, a temporary ratelimiting rule is published. |
| 2025-09-12 19:12 | The problematic changes to the Tenant API Service are reverted, and Dashboard Availability returns to 100%. IMPACT END |
Dashboard availability
The Cloudflare dashboard experienced severe impact throughout the entire incident duration.
API availability
The Cloudflare API suffered severe impact during two periods when the Tenant API Service was compromised.
How we responded
The immediate response focused on load reduction and resource allocation rather than directly fixing the underlying bug. The team implemented global rate limiting on the Tenant Service and increased Kubernetes pod capacity to improve throughput.
However, after the Tenant Service stabilized and APIs largely recovered, the team observed persistent errors from the service. They deployed a patch attempting to address these errors and restore dashboard functionality. This approach backfired: “Ultimately this change degraded service further and was quickly reverted,” creating a second outage visible in their availability graphs.
The post notes that the outage’s impact remained somewhat limited because “this was a failure in the control plane which has strict separation of concerns from the data plane.” Consequently, CDN services and cached content delivery continued uninterrupted — only users making configuration changes or accessing the dashboard experienced problems.
Going forward
Cloudflare identified improvements across two categories: impact reduction and enhanced observability.
Reducing impact
The company uses Argo Rollouts for deployments with automatic error detection and rollback capabilities. The Tenant Service hadn’t yet been migrated to this system. The post indicates “Had it been in place, we would have automatically rolled back the second Tenant Service update limiting the second outage.” This migration work, already scheduled, received increased priority.
Another concern involved the “Thundering Herd” problem: when the Tenant Service restarted, all users’ dashboards simultaneously re-authenticated, destabilizing the API again. The team released a hotfix and plans additional changes including “random delays to spread out retries and reduce contention.”
Resource allocation also required attention: “the Tenant Service was not allocated sufficient capacity to handle spikes in load like this.” Cloudflare substantially increased capacity and improved monitoring to provide proactive alerts before capacity limits are reached.
Improving visibility
During the incident, distinguishing between genuine new requests and retries proved difficult. The team observed elevated API usage but couldn’t easily identify the problematic request loop. To address this, they’re “adding changes to how we call our APIs from our dashboard to include additional information, including if the request is a retry or new request.”