By Rafael Elvira (Staff Software Engineer, Infrastructure) and Laura Nolan (Senior Staff Engineer). Published November 29, 2021.
On September 30th, 2021, Slack experienced an outage lasting 24 hours that impacted less than 1% of the online user base. This incident resulted from an attempt to enable DNSSEC, a DNS security extension required for FedRAMP Moderate compliance.
DNS and DNSSEC Background
DNS functions as a directory for the internet, translating domain names into IP addresses. The DNS protocol is inherently unsecure—anyone positioned between a client and authoritative nameserver can tamper with responses. DNSSEC adds cryptographic authentication to prevent such interference, though it does not protect the final connection between clients and recursive resolvers.
Slack uses Amazon Route 53 as its authoritative DNS server for public domains, delegating some subzones to NS1 for advanced traffic management features.
Pre-Incident Rollout
The Traffic Engineering team spent four months enabling DNSSEC signing across Slack’s domains. All critical domains except slack.com had DNSSEC successfully enabled for at least one month before the incident.
The team implemented comprehensive validation:
- External monitoring from multiple global locations using various resolvers
- DNSviz.net and Verisign DNSSEC Debugger testing
- dnschecker.org tests for resolution verification
- Hand-crafted dig command tests across multiple resolvers
Unsuccessful Attempt One (September 7th)
While attempting DNSSEC enablement on slack.com, a major U.S. ISP experienced an outage. Before publishing the DS record at the DNS registrar, the slack.com zone changes were preemptively rolled back.
Unsuccessful Attempt Two (September 8th)
The second attempt revealed that some resolvers become stricter when DNSSEC signing is enabled at authoritative nameservers, “even while signing was not enabled at the root name servers.” This strict DNS spec enforcement rejected CNAME records at zone apexes, disproportionately affecting VPN users whose providers use such resolvers.
Following this discovery, sub-delegated zones were updated to use ALIAS records instead of CNAME records.
Unsuccessful Attempt Three (September 30th)
The third attempt began with enabling DNSSEC signing on all NS1 sub-delegated slack.com zones and the Route53 authoritative zone. After three hours of successful validation testing, the team published the slack.com DS record at the DNS registrar, instructing validating resolvers to begin validating requests for slack.com.
Initial monitoring confirmed success. However, Customer Experience soon reported users seeing “ERR_NAME_NOT_RESOLVED” errors—particularly from corporate DNS resolvers and Google Public DNS (8.8.8.8).
The Cascading Crisis
The team initially removed the DS record from the DNS registrar to “stop the bleeding,” but impact persisted. After nearly an hour with no improvement, they rolled back DNSSEC signing entirely from both authoritative and delegated zones.
This rollback proved catastrophic. The team had assumed that “DS records at the .com zone were never cached,” but the .com zone actually caches the slack.com DS record for 24 hours by default (some resolvers cap this at lower values—Google uses 6 hours). Removing DNSSEC signing while cached DS records remained at resolvers caused DNSSEC validations to fail across the board.
The team’s Terraform-based DNS management missed a critical Route53 warning about disabling DNSSEC signing. Once complete, all DNSSEC-validating resolvers with cached DS records began returning SERVFAIL responses for all slack.com lookups.
The team contacted major ISPs and public DNS operators requesting cache flushes. As caches were purged and the 24-hour TTL expired, error rates gradually normalized.
Root Cause Analysis
Investigation revealed the issue centered on wildcard records. Unlike other successfully deployed DNSSEC domains, slack.com contains a wildcard record. When clients query for non-existent record types under the wildcard (such as AAAA lookups when only A records exist), Route53 was returning NSEC responses lacking the necessary additional information about existing record types.
This incomplete NSEC response caused certain recursive resolvers—particularly Google Public DNS—to infer that nothing existed at that level, preventing fallback queries for A records. The poisoned cache affected entire regional caches backing larger resolvers.
After investigation with Amazon’s Route53 team, the issue was attributed to Route53 not returning the additional information required by DNSSEC specifications for wildcard records.
A request for a record at the same level, say an AAAA lookup for app.slack.com, would correctly return a signed NSEC saying “there is no record for that.” However, without proper type bitmap information, some resolvers incorrectly interpreted this as meaning nothing existed at that domain level.
Different resolver implementations handled this ambiguity differently—Google Public DNS performed aggressive NSEC caching and type inference (per RFC 8198), while Cloudflare Public DNS did not, explaining why some resolvers showed impact while others did not.
At the time of writing, the Route53 team had deployed a fix addressing the NSEC-type bug.
Acknowledgments
The article extends gratitude to the Amazon Route53 team for confirming the issue and deploying a rapid fix, the Salesforce DNS team for expertise and support, the DNS community for cache-flushing assistance, and Slack’s Customer Experience team for connecting customers with engineering during the incident.