On 09/30/2021 between 14:00:00 UTC and 10/01/21 at 02:00:00 UTC, many of our customers experienced a variety of service degradations across all regions. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
Heroku Common Runtime dyno log tailing suffered complete unavailability during this incident. Logs were also unable to be delivered to HTTPS log drains that used a Let’s Encrypt-issued certificate. Customers using a log drain that used any other SSL certificate provider were not impacted.
Dyno metrics for applications in the Common Runtime were also impacted, as this feature relies on our internal dyno log tailing systems to report dyno usage data.
Finally, Heroku Connect customers were unable to use the Connect dashboard.
What happened?
On 9/30/2021, an older root certificate used by Let’s Encrypt expired.
Immediately after the root expired, our engineering teams received a number of alerts related to logging, metrics, and the Connect dashboard.
As we investigated, we discovered that several HTTP client libraries our systems use were using their own vendored root certificates, rather than the up-to-date root certificate bundle on the host. As a result of this outdated set of roots, the library was no longer able to make calls to services with certificates issued by Let’s Encrypt, including many of our internal services. We began to deploy a change that reconfigured the library to use the root certificate bundle from the host instead. Unfortunately, our deployment tooling was also impacted by the same issue. As a result, we were forced to rely on a slower full replacement of all of our hosts, which significantly increased our time to recovery. After manually resolving the issue with our deployment tooling, we rolled the change out across all impacted fleets. We now use the root certs on all of our hosts. Going forward, our normal OS update schedule will keep them up to date.
What will we do to mitigate problems like this in the future?
In advance of the DST Root CA X3 certificate expiry, we had audited and patched our underlying hosts and the Heroku stack images to ensure they were either (a) using an OpenSSL version that would not fail to use a valid, but shorter, certificate chain for Let’s Encrypt certificates, or (b) using an update host certificate bundle that distrusted this certificate. However, we overlooked the possibility of libraries that our services depend upon either not using or augmenting the host’s certificate bundle. Immediately following this incident, we’ve been hard at work to audit our dependencies to identify those that bundle their own root certificates to trust, and configuring them to only rely on the host’s own set of trusted root certificates.
Additionally, given how infrequently root certificates expire we’ll be working to improve our monitoring around root and intermediate certificate expiry, as well as developing game days that exercise revocation and expiry for these kinds of certificates.