Published: October 21, 2025 | Authors: Guillermo Rauch, Malte Ubl, Matthew Binshtok
At Vercel, the company’s philosophy emphasizes taking ownership for vendor-related incidents. “Customers use our services to gain velocity, reliability, and ship wonderful products,” and Vercel considers itself fully accountable regardless of which infrastructure components contribute to failures.
The incident was triggered by an unexpected AWS us-east-1 outage. Vercel acknowledges its dependence on AWS infrastructure and states: “Vercel is fully accountable for this outage,” despite the public disclosure that AWS infrastructure was the root cause. The platform operates across 19 AWS regions with terminating and securing traffic in 95 cities and 130+ global points of presence.
The outage caused two independent incidents. The first involved temporary disruption of production traffic serving when the us-east-1 data center went offline. The second stemmed from a cascading failure affecting Vercel’s feature flag provider, disrupting the control plane but not production traffic serving.
Service disruption overview
The AWS outage led to a cascading failure in Vercel’s regional cache infrastructure. Initial disruption was observed at 06:55 UTC. Traffic rerouting began at 07:15 UTC, with full restoration of serving for end users connecting from us-east-1 by 07:44 UTC.
At 07:25 UTC, the team began rerouting function invocations for customers with configured backup regions. The most severe impact occurred at 07:45 UTC when a cascading failure affected the global caching infrastructure, causing static file serving failures affecting approximately 22% of traffic. Service restoration occurred at 08:18 UTC.
For customers using only us-east-1 without multiple function region configurations, full restoration came at 09:21 UTC.
A secondary control plane disruption began at 19:20 UTC when the feature flag provider experienced major outages. The unavailability exhausted resources in the primary Kubernetes cluster. Mitigation efforts began at 19:48 UTC, with full control plane restoration around 03:00 UTC on October 21.
The disruption from the perspective of a request
The postmortem traces how a typical request to rauchg.com moved through Vercel’s infrastructure:
-
DNS lookup: Vercel DNS leveraging anycasted, intelligent DNS functioned as expected, routing to healthy endpoints.
-
CDN IP resolution: The Vercel CDN provided global acceleration via Anycast, steering traffic to the nearest healthy CDN edge region, working as expected.
-
L3-L5 protections: The Vercel Edge Network’s DDoS protection operated normally.
-
TLS handshake: The global HTTPS Terminator component, which manages tens of millions of unique domain certificates, functioned as expected.
-
L7 Firewall: Vercel Firewall’s globally distributed service operated without issues.
-
HTTP request handling: The globally replicated metadata infrastructure worked globally, though customers hitting
iad1directly experienced disruption until traffic rerouting occurred. -
CDN routing and caching: The Regional Cache component experienced a cascading failure affecting static asset and cacheable request caching. While metadata actively propagated to every region, this system experienced wider disruption.
-
Routing decisions: Different request types experienced varying impacts:
- Routing Middleware experienced initial disruption for
us-east-1region users - Function invocations saw no impact for multi-region customers; single-region
us-east-1users experienced failover - Cached HTTP responses served normally
- Static and ISR content served as expected when cached; uncached ISR content with
us-east-1origins experienced temporary disruption
- Routing Middleware experienced initial disruption for
Incident timeline
Monday, October 20, 2025
06:55 UTC: Automated alerts fire for errors across Vercel services with incidents automatically created and on-call engineers paged.
06:56 UTC: The primary on-call engineer responds and begins investigation.
06:59 UTC: Additional alerts page the primary engineer, who escalates to other on-call engineers and starts an incident call for coordinated triage.
07:07 UTC: Engineers determine a major AWS-managed services issue exists and escalate to the “Panic Rotation,” paging senior, experienced responders.
07:15 UTC: The team decides to reroute traffic away from iad1 into neighboring data centers due to serving stack failures.
07:20 UTC: Incremental traffic rerouting from iad1 begins.
07:35 UTC: Due to build machine provisioning failures, the team reroutes builds to another region, completed by 07:40 UTC.
07:40 UTC: Builds with external dependencies tied to AWS us-east-1 continue failing.
07:44 UTC: Traffic rerouting from iad1 completes.
08:16 UTC: Increased failures loading the Vercel Dashboard and elevated error rates observed in the Vercel Teams API due to elevated request latency to external dependencies and CPU starvation.
08:51 UTC: Manual service scaling applied.
09:15 UTC: The Vercel Teams API recovers; Dashboard errors cease.
09:21 UTC: Function invocations for single-region us-east-1 customers fully restored.
09:45 UTC: New Vercel Deployments using Routing Middleware or Vercel Functions with iad1 fail due to AWS API issues. This affects all projects configured with Secure Compute using iad1.
10:22 UTC: Health checks pass in the iad1 builds cluster; Secure Compute builds are rerouted there, restoring service.
11:52 UTC: Service functionality restores for APIs creating Vercel Functions and Middleware in iad1.
17:19 UTC: Small increase in failures invoking Vercel Functions in iad1 observed; simultaneous paging for Functions routing service scheduling failures.
17:24 UTC: Scheduling failures attributed to continued EC2 capacity issues in AWS us-east-1. Non-critical infrastructure provisioning reduced in iad1 to free capacity.
17:29 UTC: Configuration updated for serving stack to prefer other regions for Vercel Functions when available.
17:35 UTC: Decreased function invocation errors observed in iad1.
17:47 UTC: Another AWS API degradation causes build failures for Vercel Functions and Middleware creation in iad1.
17:49 UTC: Function invocation errors recover.
18:50 UTC: Vercel Runtime Logs service begins experiencing timeouts.
19:20 UTC: Dashboard loading failures and elevated API error responses emerge. Control Plane service containers begin restarting aggressively.
19:31 UTC: Automated alerts for elevated API errors page the primary on-call engineer; a separate incident opens.
19:41 UTC: Timeouts to Vercel’s feature flag provider identified as the underlying issue.
19:48 UTC: A change releases to reduce timeouts and mitigate failed requests to the flag provider, reducing container restarts.
21:20 UTC: An experimental change releases to degraded services in incremental batches.
22:31 UTC: After observing recovery, the experimental change rolls to all control plane degraded services.
Tuesday, October 21, 2025
00:20 UTC: Dashboard and API degradation resumes; the experimental change had not landed in 100% of services.
00:48 UTC: Missing Vercel Runtime Logs observed in several regions with 5-6 hour backlogs. A change merges to increase throughput.
02:37 UTC: All flags clients across Control Plane services reconfigured to use the experimental fix.
03:01 UTC: Another logs pipeline fix releases, successfully unblocking log processing.
03:30 UTC: Dashboard and API services fully recover.
06:21 UTC: Runtime Logs fully recover.
Future improvements
Vercel is developing a detailed technical postmortem addressing systematic mitigation of discovered failure modes. The company commits to continuing and intensifying efforts to rehearse major regional disruptions.