Incident Report: September 30 Service Disruption to BUR Data Center
Executive Summary
On Sept. 30, 2021, between 21:17 and 21:45 UTC, several of VIP’s data centers intermittently stopped forwarding traffic to the origin data center in Los Angeles. During this time, sites hosted in the Los Angeles data center experienced elevated error rates for requests originating from Africa, Europe, Eastern and Central USA. Requests from Western USA and Asia-Pacific were not affected. By 21:45 UTC the issue was resolved and error rates returned to normal levels.
Chronology of Events
| Time (UTC) | Date | Update |
|---|---|---|
| 21:17 | 30 Sep. 2021 | High error volumes detected by VIP monitoring systems. |
| 21:22 | 30 Sep. 2021 | Investigation begins into the cause of the errors. |
| 21:30 | 30 Sep. 2021 | Problem identified as a request forwarding issue to Los Angeles. |
| 21:37 | 30 Sep. 2021 | Root cause identified and configuration change rollback initiated. |
| 21:45 | 30 Sep. 2021 | Configuration rollback complete and all sites recovered. |
Business Impact
WordPress VIP applications with a Los Angeles origin experienced elevated error rates between 21:17 and 21:45 UTC on September 30, 2021 for uncached requests originating from Africa, Europe, Western and Central USA.
Root Cause Analysis
On September 20th, 2021 we deployed a configuration change that improved routing performance and reduced error rates. This involved always routing traffic from the edge to the closest origin data center and then allowing that origin data center to forward the traffic to the correct origin data center for a given site. We also configured multiple paths to each origin data center from each edge location. Here is an example for a request originating in New York to a site hosted in our Los Angeles data center:
Before:
Client → New York edge → Los Angeles
After:
Client → New York edge → Ashburn origin → Los Angeles (option #1)
Client → New York edge → Dallas origin → Los Angeles (option #2)
Client → New York edge → Los Angeles (option #3)
Even though the “before” option seems more direct, it’s less desirable because it spends more time on 3rd party provider networks. In the “before” case if there was a network problem between New York and Los Angeles, an error would likely be returned to the client. In the “After” case the time on the provider network is minimized and multiple backup paths are provided, allowing requests to be retried in the case of network problems.
Unfortunately, ten days later, a traffic anomaly triggered a software bug introduced by this optimization which caused multiple origin locations to incorrectly think that the Los Angeles origin data center was unreachable and stop forwarding traffic. When this happened, the edge health checks failed and returned 503 responses for some requests.
Remediation
The routing improvements were rolled back which restored traffic routing to the pre-September 20th state and prevented the bug from impacting live requests.
Preventative Actions
We have audited our configurations to ensure a similar problem does not exist elsewhere. Additionally, we are investigating ways to reduce the time between when the errors started and when the investigation began. In this case it was five minutes, but we would like to reduce it further.