Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region
We wanted to provide you with some additional information about the service disruption that occurred in the N. Virginia (us-east-1) Region on October 19 and 20, 2025. While the event started at 11:48 PM PDT on October 19 and ended at 2:20 PM PDT on October 20, there were three distinct periods of impact to customer applications.
DynamoDB
Between 11:48 PM PDT on October 19 and 2:40 AM PDT on October 20, customers experienced increased Amazon DynamoDB API error rates in the N. Virginia (us-east-1) Region.
The incident stemmed from a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service’s regional endpoint.
The system architecture involved two independent components: the DNS Planner monitored load balancer health and created DNS plans, while the DNS Enactor applied these plans across three availability zones. The race condition occurred when a delayed DNS Enactor applied an outdated plan that overwrote a newer plan, and the cleanup process then deleted the old plan, removing all IP addresses for the regional endpoint.
By 12:38 AM on October 20, engineers identified the DNS issue. Temporary mitigations at 1:15 AM enabled some internal service reconnection. All DNS information was restored by 2:25 AM, with customers regaining connectivity between 2:25 AM and 2:40 AM.
Amazon EC2
EC2 experienced increased API error rates, latencies, and instance launch failures from 11:48 PM PDT on October 19 through 1:50 PM PDT on October 20.
The DropletWorkflow Manager (DWFM) depends on DynamoDB for state checks. When DynamoDB became unavailable, these checks failed, causing droplet leases to time out. After DynamoDB recovery, DWFM entered “congestive collapse” as it attempted to reestablish leases simultaneously for thousands of droplets.
At 4:14 AM, engineers throttled incoming work and selectively restarted DWFM hosts. By 5:28 AM, all droplet leases were reestablished. However, Network Manager then faced delays propagating network configurations to newly launched instances, which weren’t fully resolved until 10:36 AM. Request throttles remained in place until 1:50 PM when full recovery occurred.
Network Load Balancer (NLB)
Between 5:30 AM and 2:09 PM PDT on October 20 some customers experienced increased connection errors on their NLBs in the N. Virginia (us-east-1) Region.
Health check failures occurred because the health checking subsystem brought new instances into service before their network configurations had fully propagated. This alternating pattern of failed and successful health checks increased load on the subsystem and triggered automatic AZ failover, reducing available capacity.
At 9:36 AM, engineers disabled automatic health check failovers, restoring available capacity. Automatic failover was re-enabled at 2:09 PM after EC2 recovered.
Lambda and Other Services
Lambda experienced function creation failures, processing delays, and invocation errors from 11:51 PM PDT on October 19 through 2:15 PM PDT on October 20. An internal subsystem responsible for polling SQS queues failed and required manual restoration at 4:40 AM. Starting at 7:04 AM, NLB health check failures triggered instance terminations, requiring engineers to throttle event source mappings and asynchronous invocations.
ECS, EKS, and Fargate experienced container launch failures and cluster scaling delays from 11:45 PM on October 19 through 2:20 PM on October 20.
Amazon Connect customers experienced elevated errors handling calls, chats, and cases from 11:56 PM on October 19 through 1:20 PM on October 20. After initial DynamoDB recovery, chat errors persisted until 5:00 AM. From 7:04 AM onwards, NLB issues and Lambda errors again impacted Connect services.
Security Token Service (STS) experienced API errors and latency from 11:51 PM on October 19 through 9:59 AM on October 20, with two distinct impact periods corresponding to DynamoDB and NLB issues.
IAM authentication failures affected AWS customers attempting to sign into the AWS Management Console from 11:51 PM on October 19 through 1:25 AM on October 20. This impacted both direct IAM user logins and IAM Identity Center configurations in the affected region.
Redshift experienced query processing failures and cluster creation/modification errors from 11:47 PM on October 19 through 2:21 AM on October 20 due to DynamoDB dependency. After DynamoDB recovery, some compute clusters remained impaired because credential expiration triggered replacement workflows blocked by EC2 launch impairments. Full cluster availability was restored by 4:05 AM PDT on October 21. Additionally, Amazon Redshift customers in all AWS Regions were unable to use IAM user credentials for executing queries when the N. Virginia region’s IAM API became unavailable.
AWS Support Console and API experienced outages from 11:48 PM on October 19 through 2:40 AM on October 20. Although the Support Center failed over to another region as designed, a subsystem responsible for account metadata began providing responses that prevented legitimate users from accessing the AWS Support Center.
Remediation Actions
AWS disabled the DynamoDB DNS Planner and DNS Enactor automation worldwide pending fixes. Planned improvements include fixing the race condition, adding protections against incorrect DNS plans, implementing velocity controls for NLB health check failovers, expanding EC2 scale testing, and improving throttling mechanisms in data propagation systems.
Closing Statement
We apologize for the impact this event caused our customers. While we have a strong track record of operating our services with the highest levels of availability, we know how critical our services are to our customers, their applications and end users, and their businesses.