Summary of AWS Direct Connect Event in the Tokyo (AP-NORTHEAST-1) Region

We would like to provide additional information about the AWS Direct Connect service disruption that occurred in the Tokyo (AP-NORTHEAST-1) Region on September 2, 2021. Beginning 7:30 AM JST, Direct Connect customers began to experience intermittent connectivity issues and elevated packet loss for their traffic destined towards the Tokyo Region. This was caused by the failure of a subset of network devices on one of the network layers along the network path from Direct Connect edge locations to the Datacenter network in the Tokyo Region, where customers’ Virtual Private Clouds (VPCs) reside. Customers started seeing recovery by 12:30 PM JST and by 1:42 PM JST, connectivity issues were fully resolved. All other forms of network connectivity, including traffic between Availability Zones, internet connectivity to the Region, and AWS Virtual Private Network (VPN) connectivity (which some customers use as a back-up to Direct Connect) were not impacted. Direct Connect traffic to other AWS Regions was also not impacted.

On September 2, 2021 at 7:30 AM JST, internal alarms alerted AWS engineers to elevated packet loss for Direct Connect customers connecting to the Tokyo Region. The Direct Connect service provides private connectivity between a customer’s data center and their AWS VPCs by forwarding traffic from the edge locations where AWS interconnects with customers, to the AWS Region through multiple network layers - each with many redundant network devices. These alarms identified that the impact was caused by the failure of several devices in a single layer of the Direct Connect network. While these devices were not correctly forwarding traffic, they were not being removed from the network through the normal automated processes that monitor and remove failed network devices. Our automation instead noticed a higher rate of failed devices than normal and alerted engineers to investigate and take remediation action. When engineers were alerted, they determined that there was enough redundancy at this layer and began removing the impacted devices from service so that traffic could be handled by other healthy devices. In parallel, the team investigated the cause of the failure. While the removal of additional devices provided temporary remediation, several other network devices subsequently began to experience the same failure, resulting in network congestion, connectivity issues, or elevated packet loss for Direct Connect customers. Engineers attempted several mitigations, such as resetting failed devices and slowly bringing them back into service, but the failures continued and the engineers were unable to maintain adequate healthy capacity to fully mitigate the customer impact. Engineers also looked for any recent deployments that may have triggered the failure. By 12:00 PM JST, engineers suspected that the failure may be related to a new protocol that was introduced to optimize the network’s reaction time to infrequent network convergence events and fiber cuts. This new protocol was introduced many months prior and this change had been in production since then without any issues. However, engineers suspected that the failure was related to the interaction of this new protocol and a new traffic pattern on the network devices at this layer of the Direct Connect network. Engineers started disabling this new protocol in a single Availability Zone to monitor and establish sustained recovery, while in parallel preparing the change to be deployed across the Tokyo Region. Customers started reporting recovery to their applications by 12:30 PM JST and by 1:42 PM JST affected networking devices were restored to a stable operational state and the Direct Connect service returned to normal operations.

While disabling the new protocol resolved the event, engineering teams have continued working to identify the underlying root cause. We have now confirmed that this event was caused by a latent issue within the network device operating system. This version of the operating system enables a new protocol which is used to improve the failover time of our network. The new operating system and protocol have been running successfully in production for multiple months. We use a controlled, automated, tested, and instrumented procedure for changing the operating system and introducing the new protocol to the AWS network. This procedure starts with a series of stress tests in a dedicated lab to validate the resiliency of the network device to both valid and invalid (i.e., malformed) packets. Any anomalies identified in lab testing are diagnosed, root causes identified, and remediated before the new code is released to production. Even with this comprehensive testing, it is not possible to test every traffic and packet permutation in a lab environment. Therefore, AWS uses a deployment procedure that releases network device operating system changes to production in a slow and controlled fashion. This procedure upgrades individual devices in specific places where the upgraded devices can be exposed to production traffic but where traffic can easily fail away from the upgraded devices to non-upgraded devices. During this gradual production deployment, the upgraded devices are extensively monitored for performance issues and functionality errors. This upgrade process has been used many times successfully and was followed with this most recent device operating system upgrade. The new protocol and the operating system were first deployed to production in January 2021. Over the last 8 months, this new protocol and the operating system have been gradually released to production in all AWS Regions and has been serving Direct Connect customer traffic without any indication of the latent issue. Over the last several days, engineers have been able to identify the defect in the network operating system and determined that it requires a very specific set of packet attributes and contents to trigger the issue. While these conditions are very specific and unlikely, this event was triggered by customer traffic that was able to consistently generate packets that matched this signature. We have no reason to suspect malicious intent. We have disabled the new protocol that triggered this issue in the AWS Tokyo Region. We have also developed an enhanced way to detect and remediate this issue before customer impact, as we carefully apply this change to all other AWS Regions. We are confident that there will be no additional customer impact from this issue.

We understand how critical AWS services are for our customers and many businesses in Japan, and we sincerely apologize for the impact that this event may have caused. We have a long track record of operating our services with high levels of availability and will do everything possible to maintain our customers’ trust and help them achieve the availability they need for their customers and businesses.