By Bryant Khau and Leonardo Zizzamia, May 27, 2021
Summary
On May 19, 2021, between 5:50 AM and 7:38 AM PT, Coinbase experienced connectivity issues affecting coinbase.com, mobile apps, and Coinbase Pro. The outage occurred during a significant cryptocurrency market downturn when Bitcoin dropped 25% and Ethereum fell 20%, triggering a surge in user traffic. During this period, users encountered slow load times and errors when attempting to access buying, selling, and trading features.
The Outage
Four main service categories were impacted:
The logged-out web servers became largely inaccessible, forcing unauthenticated users to error pages. The GraphQL service, which aggregates data from underlying systems, experienced timeouts and slowness that caused roughly 10% of mobile app requests to fail. Coinbase Pro’s API became partially unreachable due to database and API server strain. Additionally, the non-US card payment processing service rejected transactions, preventing international customers from purchasing cryptocurrency with cards.
Engineers organized into parallel investigation teams to address each failure independently and prioritize remediation efforts.
Root Cause Analysis
The investigation revealed four distinct technical failures:
The logged-out pages became overwhelmed when the traffic spike hit the maximum connection threshold in the Nginx router, requiring manual intervention to increase capacity. It took over 40 minutes for these servers to recover fully.
The GraphQL service struggled because its autoscaling mechanism was too slow to respond to the sudden demand increase. Once the system eventually scaled up, mobile app functionality was restored.
The Coinbase Pro database experienced elevated latency and CPU consumption, as did the market data feed API servers. Increasing database throughput and provisioning additional API servers resolved these constraints.
Payment processing failures accumulated as the queue became backlogged. Adding more queue workers allowed transactions to resume processing successfully.
Improvements
Coinbase identified that autoscaling configurations were inadequately tuned for the rapid, severe traffic spikes characteristic of cryptocurrency markets. The company plans to enhance load testing procedures to better simulate real-world scenarios and expose infrastructure weaknesses before they affect customers.
A second improvement initiative involves deploying kill switches within the client application, enabling partial system functionality during failures while engineers resolve underlying problems. The company emphasizes its commitment to supporting millions of users relying on their platform.