Report on our investigation of the 2025-10-20 incident in AWS us-east-1
By Richard Crowley | November 3, 2025
On October 20, 2025, PlanetScale experienced a significant incident in AWS us-east-1. The situation began with a DNS misconfiguration in one of PlanetScale’s service providers, which cascaded into control plane disruptions and later network connectivity challenges affecting some customer database branches.
Phase 1
Engineers received alerts at 7:13 UTC when production tests started failing. The investigation revealed a near-total control plane outage centered on the database branch management service in us-east-1. The root cause chain extended through their secret-distribution system to Amazon S3 and AWS STS, which experienced impacts from an Amazon DynamoDB outage.
Despite control plane problems, no database branches lost capacity or connectivity during this phase. However, the PlanetScale dashboard became intermittently unavailable, SSO users faced login difficulties, and the status page could not be updated. Service was restored at 9:30 UTC once upstream providers recovered.
Phase 2
At 10:05 UTC, responders discovered inability to launch new EC2 instances in us-east-1. While existing database servers remained operational, requests to create or resize branches accumulated in queues. This posed significant risk as East Coast customers faced their Monday workday with reduced vtgate capacity.
The team implemented several measures: temporarily blocking new database creation in us-east-1, redirecting to us-east-2; delaying backups; advising customers to shed load; and pausing termination of aged instances. Critically, they temporarily changed how they schedule vtgate processes by bin-packing them more densely to maximize available capacity.
Network partitions emerged around 14:30 UTC, causing some customer queries to fail. These partitions gradually resolved between 18:30 and 19:30 UTC. After healing, some processes required manual restarts. The incident was declared resolved at 20:32 UTC.
Reflecting on Resilience
The company credits strong separation between control and data planes for limiting impact. Network partitions remain their most challenging failure mode, prompting consideration of better utilization of AWS us-east-1’s six availability zones for improved fault tolerance.