Cloudflare 1.1.1.1 incident on June 27, 2024
Authors: Bryton Herdes, Mingwei Zhang, Tanner Ryan
Introduction
On June 27, 2024, Cloudflare’s 1.1.1.1 DNS resolver experienced an outage affecting a small global user population. The incident resulted from a combination of BGP hijacking and route leakage. Although Cloudflare had adopted RPKI (Resource Public Key Infrastructure) for route origin validation, the 1.1.1.1/32 address was hijacked by ELETRONET S.A. (AS267613) and accepted by multiple networks, including at least one Tier 1 provider. This caused unreachability across more than 300 networks spanning 70 countries, though overall user impact remained below 1% in most regions.
Background
Cloudflare launched the 1.1.1.1 public DNS resolver in 2018, which has since become one of the most widely used free DNS services. The address’s recognizable nature and historical use as a test IP in lab environments have created operational challenges. The service faces recurring issues with BGP misrouting, including hijacks and route leaks.
BGP Hijacks
BGP hijacking occurs when unauthorized networks advertise IP prefixes they don’t control. In the 1.1.1.1 context, if a network like FooBar Networks were to announce the 1.1.1.1/32 address, routing algorithms would prefer this more specific route over Cloudflare’s broader 1.1.1.0/24 announcement due to Longest Prefix Matching (LPM). This would redirect traffic away from legitimate Cloudflare servers.
BGP Route Leaks
Route leaks happen when a network becomes an upstream BGP announcer for networks it shouldn’t serve. For example, a customer might forward routes from one provider to another, creating a “type 1 leak” per RFC 7908. Even with 13,000+ global interconnections, Cloudflare can experience issues when leaked routes receive higher BGP local-preference than direct announcements. This occurs because providers assign higher priority to paid transit connections versus peer relationships.
Incident Timeline and Impact
June 27, 2024 — 18:51 UTC: AS267613 begins announcing 1.1.1.1/32 with AS267613 as origin
June 27, 2024 — 18:52 UTC: AS262504 (Nova) leaks 1.1.1.0/24 upstream to AS1031 (Peer-1 Global Internet Exchange) with AS path “1031 262504 267613 13335”
June 27, 2024 — 18:52 UTC: AS1031 propagates the 1.1.1.0/24 leak to Internet Exchange peers and route-servers
June 27, 2024 — 18:52 UTC: A Tier 1 provider accepts the 1.1.1.1/32 announcement as a RTBH (Remote Triggered Blackhole) route, blackholing traffic for all downstream customers
June 27, 2024 — 20:03 UTC: Cloudflare identifies internal incident regarding 1.1.1.1 reachability issues
June 27, 2024 — 20:08 UTC: Cloudflare disables peering with AS267613 at one location receiving traffic toward 1.1.1.0/24
June 27, 2024 — 20:08 UTC: Cloudflare team contacts AS267613 about the incident
June 27, 2024 — 20:10 UTC: AS262504 leaks 1.1.1.0/24 with new AS path “262504 53072 7738 13335,” which AS1031 redistributes; traffic reaches Cloudflare with high latency
June 27, 2024 — 20:17 UTC: Cloudflare engages AS262504 regarding the route leak
June 27, 2024 — 21:56 UTC: Cloudflare disables second peering point with AS267613
June 27, 2024 — 22:16 UTC: AS262504 leaks 1.1.1.0/24 again; some requests route through São Paulo with higher latency, but 1.1.1.1/32 hijacking appears resolved
June 28, 2024 — 02:28 UTC: AS262504 fully resolves the route leak
Impact manifested in two ways: complete unreachability or elevated latency. Traffic destined for 1.1.1.1 from Germany and the United States was being routed to Brazilian data centers instead of optimal local endpoints. Gaps between traffic spikes represented periods of blackholing, while spikes indicated successful delivery with degraded performance.
Technical Description of the Error
Normally, 1.1.1.1 requests route to the nearest data center via BGP anycast. During the incident, AS267613 advertised 1.1.1.1/32 while AS262504 leaked 1.1.1.0/24 upstream, fundamentally disrupting normal routing patterns.
Using the monocle tool to search public route collectors revealed multiple networks received the 1.1.1.1/32 announcement from AS267613 around incident onset. Though the Default-Free Zone (DFZ) typically accepts only /24 prefixes or larger, multiple networks forwarded the 1.1.1.1/32 route, resulting in traffic blackholing that never reached Cloudflare Points of Presence.
The 1.1.1.1/32 origination by AS267613 constituted a clear BGP route hijack. The Route Origin Authorization (ROA) only permits origin AS13335 (Cloudflare) with maximum prefix length /24. Cloudflare’s own systems detected and rejected these 1.1.1.1/32 announcements as both RPKI Invalid and DFZ Invalid due to incorrect origin AS and invalid prefix length.
Multiple networks accepted an RPKI Invalid route, compounding the problem. Critically, one Tier-1 transit provider accepted 1.1.1.1/32 from AS267613 as a Remote-Triggered Blackhole (RTBH) route, discarding all customer traffic destined for the address. This approach, typically used for DDoS mitigation, was unauthorized — only Cloudflare should control blackhole actions for its own addresses.
Regarding the 1.1.1.0/24 leak, the problematic AS path “199524 1031 262504 267613 13335” shows AS267613 as a peer of Cloudflare (AS13335), yet they shared announcements with upstreams. AS262504, a customer of AS267613, further propagated the prefix upstream to AS1031. This violated normal customer cone relationships. AS1031 then redistributed the announcement to peers globally, amplifying impact despite lacking authorization.
AS262504 functioned as the leaking network, while AS1031 accepted and widely distributed the hijack and leak. AS1031 appears to perform minimal customer BGP filtering, instead automatically redistributing matching announcements based on AS adjacency alone. During most of the incident, the leaked path directed requests into AS267613, which discarded 1.1.1.1/32 traffic internally.
Remediation and Follow-Up Steps
BGP Hijacks
RPKI Origin Validation
RPKI recently achieved 50% deployment in terms of prefixes signed via Route Origin Authorization (ROA). While RPKI constrains hijacked prefix propagation, widespread enforcement across large networks remains essential. During the 1.1.1.1/32 hijack, numerous networks accepted and utilized the unauthorized route for traffic forwarding.
RPKI and Remote-Triggered Blackholing
Significant incident impact stemmed from a Tier 1 provider accepting 1.1.1.1/32 as a blackhole route from non-Cloudflare third parties. RTBH serves legitimate purposes against large DDoS attacks, but current BGP filtering relies heavily on AS-SET objects from Internet Routing Registries — insufficient authorization mechanisms. Direct ROA-based RTBH filtering would require thousands of entries, creating vulnerability to spoofed /32 announcements.
A potential solution involves RPKI Discard Origin Authorization (DOA) objects, which would restrict blackhole authorization to legitimate owners. If Tier-1 providers validated RTBH requests against DOA objects, AS267613’s unauthorized 1.1.1.1/32 blackhole attempt would have been rejected.
BGP Best Practices
Implementing MANRS (Mutually Agreed Norms for Routing Security) guidelines and rejecting IPv4 prefixes longer than /24 in the DFZ would have substantially reduced incident impact. Standard BGP policies should reject invalid prefix lengths across the Internet.
BGP Route Leaks
Route Leak Detection
Cloudflare expanded data sources for its route leak detection system and is incorporating real-time data to enable faster incident response.
ASPA for BGP
Cloudflare continues advocating RPKI adoption alongside Autonomous System Provider Authorization (ASPA) objects for AS Path-based leak prevention. Rather than signing prefixes with authorized origins, ASPA signs autonomous systems with lists of permitted provider networks. In the incident scenario, if AS267613 had signed authorized providers and AS1031 validated paths against that list, the leak would have been detected.
Alternative Approaches
RFC 9234 employs peer roles within BGP capabilities and attributes. The “Only-To-Customer” (OTC) attribute prevents upstream prefix propagation like the 1.1.1.0/24 leak. However, this requires comprehensive BGP configuration and vendor support development.
All route leak solutions demand cooperation among Internet network operators.
Conclusion
Cloudflare’s 1.1.1.1 DNS resolver fell victim to simultaneous BGP hijack and route leak events orchestrated by external networks beyond direct Cloudflare control. The company intends to accelerate impact detection internally and support Internet community efforts to minimize future user disruption.
Long-term strategies focus on broader RPKI-based origin validation and AS path validation adoption. Origin validation shows widespread deployment among major networks, while AS path validation remains in IETF draft phases. Users can verify ISP RPKI enforcement by visiting isbgpsafeyet.com.