Cloudflare 1.1.1.1 incident on July 14, 2025
Authors: Ash Pallarito, Joe Abley
Overview
On July 14, 2025, Cloudflare experienced a global outage affecting its 1.1.1.1 public DNS Resolver service. The incident lasted 62 minutes, from 21:52 UTC to 22:54 UTC, impacting the majority of users worldwide who rely on the resolver service.
Background
Cloudflare introduced the 1.1.1.1 public DNS Resolver in 2018, making it one of the most popular free DNS services available. The service operates using anycast routing, which allows traffic to be served from multiple global locations, enhancing capacity and performance. However, this architecture means that problems with route advertisements can result in global outages.
Cloudflare manages different service requirements through its Data Localization Suite (DLS), which ensures specific services’ IP addresses are Internet-reachable only in designated regions to meet compliance needs.
Incident Timeline
| Time (UTC) | Event |
|---|---|
| 2025-06-06 17:38 | Configuration error introduced for a pre-production DLS service; 1.1.1.1 Resolver prefixes inadvertently included with new service prefixes. No immediate impact as service not yet in production. |
| 2025-07-14 21:48 | Configuration change for the same DLS service triggered global refresh; 1.1.1.1 prefixes began withdrawing from production data centers. |
| 2025-07-14 21:52 | DNS traffic to 1.1.1.1 Resolver drops globally. |
| 2025-07-14 21:54 | BGP origin hijack of 1.1.1.0/24 by Tata Communications India (AS4755) exposed — not a cause of failure, but visible due to prefix withdrawal. |
| 2025-07-14 22:01 | Internal health alerts fire; incident declared. |
| 2025-07-14 22:20 | Configuration reverted to restore previous state. |
| 2025-07-14 22:54 | Service fully restored; resolver alerts cleared. |
| 2025-07-14 22:55 | Incident resolved. |
Impact
The outage affected all traffic destined for the 1.1.1.1 Resolver service across multiple IP address ranges:
- 1.1.1.0/24, 1.0.0.0/24
- 2606:4700:4700::/48
- 162.159.36.0/24, 162.159.46.0/24
- 172.64.36.0/24, 172.64.37.0/24, 172.64.100.0/24, 172.64.101.0/24
- 2606:54c1:13::/48
- 2a06:98c1:54::/48
Query rates dropped immediately and significantly across UDP, TCP, and DNS over TLS (DoT) protocols. DNS-over-HTTPS (DoH) traffic remained relatively stable since most users access the service through the domain cloudflare-dns.com rather than by IP address.
When the initial fix was applied at 22:20 UTC, a spike in traffic appeared as clients retried queries. Full routing restoration across all locations occurred by 22:54 UTC.
Technical Description
Root Cause
A configuration change on June 6 for a pre-production DLS service accidentally included a reference to the 1.1.1.1 Resolver service and its associated IP prefixes. This error remained dormant since the DLS service wasn’t yet in production.
On July 14, adding an offline data center location to the DLS service topology triggered a global configuration refresh. This refresh reduced the 1.1.1.1 Resolver’s service topology from all locations to a single offline location, causing immediate worldwide withdrawal of all 1.1.1.1 prefixes.
Investigation Findings
Cloudflare manages service topologies using both legacy and strategic systems that are synchronized. The legacy approach relies on hardcoding explicit lists of data center locations attached to particular prefixes — a method prone to errors during infrastructure expansion.
According to the post, “The legacy approach of hard-coding explicit lists of data center locations…has proved error-prone, since (for example) bringing a new data center online requires many different lists to be updated and synced consistently.”
The newer system describes topologies without hardcoding IP addresses and supports staged deployment with health monitoring. During the transition between systems, both must be maintained and synchronized, creating complexity.
BGP Hijack Observation
While investigating route withdrawals, Cloudflare discovered that Tata Communications India (AS4755) had begun advertising 1.1.1.0/24. However, this was explicitly not the cause of the outage — rather, it became visible once Cloudflare withdrew its own routes.
Service Restoration
The fix involved reverting to the previous configuration at 22:20 UTC. Initial restoration brought traffic to approximately 77% of pre-incident levels. However, about 23% of edge servers had automatically reconfigured to remove required IP bindings during the topology change.
Restoring these configurations normally requires several hours through a progressive rollout to prevent additional impact. Given the incident severity, Cloudflare accelerated the rollout after verifying changes in test locations, achieving normal traffic levels by 22:54 UTC.
Remediation and Follow-up Steps
Cloudflare identified several measures to prevent similar incidents:
Staging Addressing Deployments: Legacy systems will be deprecated to enable modern progressive deployment methodologies with staged rollouts and health-mediated automatic rollbacks.
Deprecating Legacy Systems: Cloudflare will accelerate migration away from risky deployment methods currently required by legacy components, moving to systems with higher documentation and test coverage standards.
Conclusion
The outage resulted from an internal configuration error, not external attacks or BGP hijacking. Cloudflare acknowledges the disruption caused and is implementing improvements to enhance stability and prevent recurrence.