In the late evening of January 26th, Facebook had its largest outage in more than four years. For approximately an hour both Facebook and Instagram (owned by Facebook) were completely down, along with numerous other affected sites such as Tinder and HipChat.
Facebook’s own post mortem and statements suggested the outage occurred “after we introduced a change that affected our configuration systems.”
Now, three days later a lot has been written about the outage, much of it only partially accurate. Let’s take the Facebook post mortem as a starting point and see how the outage unfolded.
Facebook Data Center and Network Architecture
Facebook maintains data centers in Prineville OR, Forest City NC and Luleå, Sweden (with Altoona IA ramping up). DNS-based global load balancing, based on “internet topology, user latency, user bandwidth, compute cluster load/availability/performance,” distributes traffic from visitors around the world to these data centers. Traffic is served by Facebook’s fiber network which spans East Asia, Australia, U.S. and Western Europe as well as transit providers for traffic from other locations, such as Africa, South America and India.
Under normal conditions, traffic to Facebook flows through two primary data center clusters, with Prineville OR and Forest City NC. Facebook’s network flows through their backbone to aggregation switches, cluster switches and through the rack to the servers. The “4-post” architecture is used at the Forest City data center. Facebook’s next generation architecture, called Fabric, was set to roll out in Altoona IA, though no customer-facing production traffic was flowing there yet at the time.
Facebook also has several other services with distinct architectures. Instagram is served primarily from Amazon Web Services US-East region. WhatsApp utilizes a SoftLayer data center in Northern Virginia.
The January 26th Facebook Outage
At 10:10pm Pacific on January 26th, TCP connections to Facebook timed out, as their engineering team likely shut off traffic. There was a drop in availability caused by TCP connection problems and near complete packet loss.
During the outage, packets were dropped within Facebook’s network, likely by an update to their Access Control List (ACL). Traffic timed out at the aggregation switches inside Prineville, before ever reaching the racks. At approximately 11:05pm, Facebook began allowing TCP connections again, with full service restored around 11:20pm.
How Instagram Fared
Instagram, another Facebook-owned service affected by the outage, had a somewhat different experience. Although the service was unavailable, throwing 503 “service unavailable” errors, TCP connections were completed to the web servers. This is likely due to the fact that Instagram is hosted on Amazon Web Services and did not require the same sort of network isolation to tackle the configuration issues. Instagram did not terminate TCP connections, instead it replied with 503 errors, even while the network paths looked healthy.
Monitoring Outages
The outage prompted broader interest in monitoring major cloud services, drawing comparisons to previous outages such as the Craigslist DNS hijack, the Time Warner Cable outage and a GitHub DDoS incident.