By Coinbase, Published December 21, 2021
Summary
Between 4:00 pm and approximately 5:36 pm PT on Tuesday, November 23rd, Coinbase experienced an outage across most production systems. During this outage, users were unable to access Coinbase using websites and apps, and therefore were unable to use their products. This post describes what occurred and the causes, and discusses how the company plans to avoid such problems in the future.
The Incident
On November 23rd, 2021, at 4:00pm PT (Nov 24, 2021 00:00 UTC) an SSL certificate for an internal hostname in one of Coinbase’s Amazon Web Services (AWS) accounts expired. The expired SSL certificate was used by many internal load balancers which caused a majority of inter-service communications to fail. Due to the API routing layer connecting to backend services via subdomains of this internal hostname, about 90% of incoming API traffic returned errors.
Error rates returned to normal once load balancers were migrated to a valid certificate.
Context: Certificates at Coinbase
Certificates for public hostnames like coinbase.com are managed and provisioned by Cloudflare. For certificates for internal hostnames used to route traffic between backend services, Coinbase historically leveraged AWS IAM Server Certificates.
IAM Server Certificates required manual generation outside AWS and uploading via API call. Last year, the infrastructure team migrated to AWS Certificate Manager (ACM). ACM solves security concerns because AWS generates both public and private certificate components within ACM and stores the encrypted version in IAM. Only connected services like Cloudfront and Elastic Load Balancers receive certificate access. Denying the acm:ExportCertificate permission to all AWS IAM Roles ensures certificates cannot be exported.
ACM also automatically renews certificates before expiration. Given that ACM certificates are supposed to renew and a migration occurred, how did this happen?
Root Cause Analysis
Incident responders quickly noticed that the expired certificate was an IAM Server Certificate. This was unexpected because the ACM migration had been widely publicized in engineering communication channels; thus the company had been operating under the assumption of running exclusively on ACM certificates.
As later discovered, one certificate migration did not proceed as planned; the group of engineers working on the migration uploaded a new IAM certificate and postponed the rest of the migration. Unfortunately, the delay was not as widely communicated as it should have been and changes to team structure and personnel resulted in the project being incorrectly assumed complete.
Alerts were being sent to an email distribution group that consisted of only two individuals. This group was originally larger, but shrank with the departure of team members and was never sufficiently repopulated as new folks joined the team.
In short, the critical certificate was allowed to expire due to three factors:
- The IAM to ACM migration was incomplete.
- Expiration alerts were only being sent via email and were filtered or ignored.
- Only two individuals were on the email distribution list.
Resolution & Improvements
To resolve the incident, Coinbase migrated all load balancers using the expired IAM cert to the existing auto-renewing ACM cert that had been provisioned as part of the original migration plan. This took longer than desired due to the number of load balancers involved and the company’s cautiousness in defining, testing, and applying the required infrastructure changes.
To ensure this issue does not recur, the following steps have been taken:
- Completed the migration to ACM, no longer using IAM Server Certificates and deleting legacy certificates to reduce noise.
- Added automated monitoring connected to the alerting and paging system to augment email alerts. These will page on impending expiration as well as when ACM certificates drop out of auto-renewal eligibility.
- Added a permanent group-alias to the email distribution list. This group is automatically updated as employees join and leave the company.
- Built a repository of incident remediation operations in order to reduce time to define, test and apply new changes.