On April 22, 2022 from 00:48am until 2:22am UTC, the logging system experienced degraded performance, which caused interruptions in the delivery of application and routing logs for customers on the Common Runtime, Private Spaces, and Salesforce Functions.
Then, between April 25, 2022 11:05pm and April 26, 2022 7:15am UTC, the Heroku Engineering teams became aware of an issue that impacted log delivery, dyno booting, builds, and releases which impacted all customers in the Common Runtime and Private Spaces. During this time, log delivery was degraded, new dynos did not boot, and builds and releases may not have been completed successfully. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
All customers in the Common Runtime, Private Spaces, and Salesforce Functions were impacted at various times during the incident periods.
What happened?
On April 22, 2022, an infrastructure deployment created a high number of database connections within the logging system, causing degradation of log delivery. Remediation efforts restored logging services to normal levels. Then, on April 25, 2022, an unrelated infrastructure change caused the logging fleet to restart, which is a normal activity, however, remediation efforts from the prior incident caused a slow degradation that manifested in longer than normal boot times for new dyno instances, which eventually hit our timeouts and led to restart cycle loops and incomplete startup attempts. This also caused logging components to stop responding to requests, and as a result, new log drains were not able to sync to the logging infrastructure, thus further extending the impact to builds and releases.
The incident was remediated by adjusting state synchronization streams to release locks more frequently and temporarily blocking logging data synchronization requests to the logging systems so that they could come back online in a controlled manner, rather than as a thundering herd all at once. Once all logging systems were restored, changes were made to create database isolation between the logging API and state synchronization services to prevent this problem from occurring again.
What will we do to mitigate problems like this in the future?
Planned improvements to the design of the logging system will decouple components such that logging failures will not impact dyno booting, builds, or releases.