When DNSSEC goes wrong: how we responded to the .de TLD outage

Authors: Sebastiaan Neuteboom, Christian Elmerot, Max Worsley

Overview

On May 5, 2026, at approximately 19:30 UTC, DENIC (the registry operator for Germany’s .de country-code top-level domain) began publishing invalid DNSSEC signatures. This caused validating DNS resolvers, including Cloudflare’s 1.1.1.1 service, to reject the responses and return SERVFAIL errors. Since .de ranks among the most widely queried TLDs globally, the misconfiguration potentially affected millions of domains.

How DNSSEC Works

DNSSEC adds cryptographic authentication to DNS through digital signatures called RRSIG records. Unlike encryption protocols such as DNS over TLS or HTTPS, DNSSEC focuses on integrity verification rather than privacy — records remain visible but their authenticity can be proven.

The system operates on a chain of trust starting from the root zone. Each zone delegates trust to child zones using Delegation Signer (DS) records containing cryptographic hashes of child zone public keys. When validating example.de, resolvers verify the entire chain: root trusts .de, and .de trusts example.de. A break anywhere in this chain causes validation failure for everything beneath it.

Zones typically employ two key types: the Zone Signing Key (ZSK) for signing zone records, and the Key Signing Key (KSK) for signing the ZSK itself. The KSK’s public key anchors the chain of trust through the parent zone’s DS record. While rotating a ZSK is relatively straightforward, rotating a KSK requires parent zone DS record updates, often necessitating registrar or registry coordination.

During key rotation, a critical vulnerability exists: if published signatures use a key that resolvers cannot verify against published DNSKEY records — due to signing failures, timing issues, or incomplete key distribution — resolvers must reject responses and return SERVFAIL.

What We Saw

Beginning at 19:30 UTC on May 5, DENIC started publishing incorrect DNSSEC signatures for the .de zone. Validating resolvers, including 1.1.1.1, were required by specification to reject these records and return SERVFAIL.

Response code data from 1.1.1.1 showed an immediate SERVFAIL spike at 19:30 UTC, followed by a steady climb over three hours as cached records expired. Each expiration forced resolvers to fetch fresh records from DENIC, which contained broken signatures, triggering additional failures.

Query volume increased substantially during the incident — typical behavior during DNS outages as clients retry failed requests multiple times, amplifying raw numbers beyond actual user impact. Notably, the NOERROR response rate remained relatively stable throughout, a phenomenon explained by “serve stale” functionality.

Serve Stale

Recursive resolvers cache records from authoritative nameservers for each record’s Time-to-Live (TTL) duration. Once cached, the resolver serves records directly without contacting authoritative servers again. Upon TTL expiration, resolvers fetch fresh copies.

During the outage, newly requested records encountered broken DNSSEC signatures. However, many .de records cached before the incident remained available. Rather than immediately discarding expired records and returning SERVFAIL, 1.1.1.1 continued serving them — a practice called “serving stale.”

The 1.1.1.1 resolver implements RFC 8767, which formalizes this behavior. When upstream resolution fails, resolvers may continue serving expired cached records instead of returning errors, significantly cushioning upstream outage impacts by providing additional response time for operator intervention.

Examination of response codes excluding stale-served responses reveals the NOERROR rate dropped steadily from 19:30 onward, representing queries receiving functional answers solely because records remained cached.

Our Mitigation

While largely beyond direct control and with serve stale providing substantial protection, legitimate user impact persisted. Several actions improved the situation.

Negative Trust Anchors

RFC 7646 defines Negative Trust Anchors (NTAs), explicit exceptions instructing resolvers to treat specific zones as unsigned, bypassing validation. These mechanisms exist specifically for TLD operator misconfiguration scenarios. When broken signatures appear at the TLD level, every DNSSEC-validating resolver returns SERVFAIL for all underlying domains — not due to domain-specific issues but parent zone misconfiguration. Continuing to return SERVFAIL provides no security value when the failure is already widely known and publicly documented.

Implementation Details

At the time of the incident, Cloudflare’s Big Pineapple resolver (powering 1.1.1.1 and related services) lacked a native NTA mechanism. Instead, we deployed an existing override rule marking .de as insecure, effectively treating all .de queries as DNSSEC-unsigned — functionally equivalent to an NTA, though not formally RFC-defined.

This represents a deliberate security tradeoff. Bypassing DNSSEC validation renders .de domains vulnerable to genuine attacks during the incident duration. However, we assessed this acceptable because the signing failure was widespread, publicly confirmed, and equally affected every validating resolver globally. As expressed internally: “There is no user of 1.1.1.1 resolving a .de name right now who would prefer a SERVFAIL over an unvalidated response.”

We deployed mitigation at 22:17 UTC, ending impact for 1.1.1.1 users. We communicated this action with fellow DNS operators through DNS-OARC Mattermost channels.

Origin Resolution Mitigations

Cloudflare maintains a separate internal resolver for origin resolution, distinct from public 1.1.1.1. We applied similar .de NTA treatment to restore connectivity for CDN customers with .de origin names affected by the outage.

Extended DNS Errors

Prior to mitigation, uncached queries received SERVFAIL responses including Extended DNS Error (EDE) codes (RFC 8914), providing clients with additional failure context.

Some resolvers correctly returned EDE 6 (DNSSEC Bogus) with descriptive messages identifying broken signatures:

“EDE: 6 (DNSSEC Bogus): RRSIG with malformed signature found for example.de/nsec3 (keytag=33834)”

However, 1.1.1.1 returned EDE 22 (No Reachable Authority), superficially suggesting upstream connectivity problems rather than DNSSEC validation failures.

The root cause is a bug in DNSSEC EDE code propagation. When the trust chain verifier detects bogus signatures and creates DNSSEC Bogus EDE codes, these never reach response layers. Instead, the outer resolver layer observes recursive resolution problems lacking error codes and defaults to “No Reachable Authority,” obscuring underlying DNSSEC causes.

Cloudflare recognizes this inadequately serves 1.1.1.1 users and will address DNSSEC error surfacing in responses.

Is This a Failure of DNSSEC as a Technology?

An outage of this magnitude and the applied mitigations might suggest DNSSEC has failed as a technology. However, misconfiguration of any technology creates user-facing risks. Leaving critical submarine fiber cables exposed to shark damage doesn’t invalidate underwater cables’ critical Internet role — it merely reveals inadequate protection strategies. Similarly, DNSSEC serves critical functions ensuring DNS answer reliability against tampering. The incident highlights protection gaps rather than technology failure.

#HugOps

Serious incidents unfortunately occur for infrastructure operators at scale. The DNS community characteristically supports affected operators during such events.

These situations underscore why operator relationships matter. DNS decentralization means no single organization controls all systems; reliable operation depends on mutual trust and open communication between registries, resolvers, and broader community members. Forums like DNS-OARC provide shared channels where operators quickly coordinate across organizational boundaries when failures occur.

DENIC published a statement noting: “The outage is linked to a routine, scheduled key rollover. During this process, non-validatable signatures were generated and distributed. As a precautionary measure, future rollovers have been suspended until the exact technical causes have been identified.”

The community anticipates additional details when DENIC’s analysis concludes.

Takeaways from This Incident

This incident reveals a structural DNS hierarchy reality: TLD-level registry failures simultaneously affect every underlying domain regardless of hosting location or resolver choice. This isn’t unique to DNSSEC; unreachable TLD nameservers produce identical effects. The hierarchy enabling global DNS functionality also propagates top-level failures downward.

No simple solution exists. Industry response speed and consistency matter most. During this incident, Internet resolver operators independently deployed Negative Trust Anchors within one hour, restoring resolution while DENIC addressed zone issues. Operational practices, industry communication channels like DNS-OARC, and features like serve stale all reduce impact, though they cannot eliminate underlying dependencies.

Cloudflare identified internal improvement areas. The organization will enhance EDE error handling to better surface DNSSEC-specific failures.

Cloudflare anticipates DENIC’s post-incident report and appreciates their transparency throughout the event.