Post Mortem: The Ugly, the Bad & the Good

Author: Matthew Prince

Overview

CloudFlare experienced a significant DNS infrastructure outage on February 24, 2012, around 07:30 GMT while deploying an update designed to accelerate DNS propagation. The incident affected multiple data centers and resulted in website unavailability for some users.

The Ugly

During the deployment across 14 data centers, a minor issue emerged affecting data transfers from the primary DNS database. While diagnosing this problem, the primary database was accidentally deleted. The new DNS system’s rapid propagation feature meant the deletion spread quickly across 10 active data centers. This caused recursive DNS lookups to return invalid results, taking affected sites offline.

The Bad

Recovery took approximately 5 minutes to identify the issue, retrieve backups, and deploy the correction. However, several complications extended the outage:

• DNS caching delays: ISP recursive DNS servers cached the bad results. Without easy cache-flushing mechanisms, users experienced prolonged outages even after CloudFlare restored correct data.

• Partial update failures: Two data centers failed to accept the corrected DNS files properly, likely due to the volume of records involved. These locations returned incorrect results until approximately 08:10 GMT, affecting European and Asian visitors particularly.

• Compounding factors: Simultaneous mid-sized DDoS attacks against customers reduced defensive capacity with fewer operational data centers, causing minimal 500 errors. Additionally, erroneous TLD entries for domains like co.nz caused extended outages for affected records.

The Good

CloudFlare implemented immediate safeguards to prevent recurrence. The engineering, operations, and support teams mobilized to assist affected customers and build preventative measures.

The new DNS infrastructure, despite the incident, offers significant benefits:

• Instant propagation: DNS changes now deploy instantly instead of taking approximately one minute
• Enhanced security: Added hardening against DNS-directed DDoS attacks
• Performance leadership: CloudFlare claims fastest global authoritative DNS updates

Matthew Prince acknowledged the incident as unacceptable and apologized to affected users, noting this represented the second network-wide outage since the previous incident over a year prior.