Dynos Unable to Start

Updates (chronological)

2014-06-23T18:06:07Z — Elevated Error Rates (issue) We are seeing elevated error rates across the platform. We are investigating.

2014-06-23T18:14:37Z — update Due to our scheduled maintenance this morning, dynos are currently unable to start or be restarted. We are working to address the issue.

2014-06-23T18:36:51Z — update We are in progress of rolling out a fix and restarting affected software. Dynos that need to start are beginning to come back online, but progress is slow. We are working to speed up the process and address any anomalies that have occurred as a result of the outage.

API and deployment services are in lockdown while we restore availability.

2014-06-23T18:51:11Z — update The fix has been rolled out and many dynos and applications are now up. We’re working to remediate remaining dynos, bring error rates back to normal, and resume API and deployment services.

2014-06-23T19:14:46Z — update We are continuing to work to address elevated error rates, such as H99s.

To help with platform stability, API and deployment services remain unavailable.

2014-06-23T19:35:52Z — update We’ve identified and addressed the cause of the remaining errors. Error rates are returning to normal levels.

2014-06-23T20:00:36Z — monitoring We’ve opened up API access and deployment services.

Error rates have returned to normal. We are monitoring error rates and overall platform health.

2014-06-23T20:13:26Z — monitoring We are addressing an issue with starting or restarting PX dynos.

2014-06-23T20:30:00Z — update We’ve addressed the issue with PX dynos.

We are continuing to carefully monitor the platform for any further anomalies.

2014-06-23T21:14:00Z — resolved All error rates have returned to normal.

We understand this incident had significant impact to our customers. We are truly sorry for the impact this has had to your businesses. We will be performing a retrospective and public follow-up as soon as possible.

2014-06-25T22:49:59Z — follow-up (postmortem)

On June 23rd, from 18:14 to 19:43 UTC, we experienced an outage for some of our customers, significantly impacting your businesses and your users. We’re sorry that we caused this downtime and for the impact it had on your businesses.

We originally scheduled a maintenance window on June 23rd from 17:00 UTC to 19:00 UTC to roll the credentials on Redis servers used by runtime components that run dynos. Due to a process error, we had to put our US API in maintenance mode from 18:14 UTC to 19:43 UTC. During this time, none of our customers were able to deploy. In addition, customers in the US were unable to scale, stop or restart dynos, and some running dynos were affected by the inability to route HTTP traffic to them.

Here is some additional detail about what happened and steps we are taking to mitigate future outages of a similar nature.

Who was affected?

All of our customers were prevented from deploying and scaling dynos while the API was locked down between 18:14 UTC and 19:43 UTC. In total 3.4% of apps receiving HTTP traffic experienced H99s between 17:00 UTC and 21:00 UTC due to issues related to the incident.

What happened?

The platform component responsible for running and monitoring dynos on our large fleet of runtime instances communicates with our runtime instances using four Redis servers. If one Redis server fails, the dyno manager maintains communication with our runtime agents using the other three servers.

On June 23rd we performed a credential roll on these Redis servers in our US cloud during a two hour scheduled maintenance window. Because we operate a service used globally, there is a less-than 10% difference in usage between so-called “peak hours” and “non-peak” hours. We scheduled maintenance for this time because it was not a peak time, but moreso because this period has high coverage from relevant engineering teams, should issues arise. By performing maintenance during this period, we were able to react more quickly and muster those teams within seconds.

The correct procedure is to change the credentials incrementally and update the components involved between each iteration. When we change the credentials on a Redis server, the dyno manager and the runtime agents talking to it should failover to the unchanged servers until they’ve been reconfigured to use the new credentials. Due to incorrect operational documentation, we failed to perform this procedure correctly, and changed the credentials on all four Redis servers at the same time.

Before realizing this issue we started a rolling restart of our runtime agents. This caused a loss of communication between the runtime agents on the updated instances and the dyno manager. While dynos kept running, the dyno manager stopped receiving status updates and was unable to start new dynos, restart existing dynos or respond to dyno crashes and other types of failures. This caused some apps to experience routing issues when dynos crashed.

Finally, poor communication between operational staff and support staff led to delays in posting updates to our status site and responding to support tickets from our customers.

What did we do to prevent wider impact?

When the credentials were changed and we were restarting our runtime agents, old and new runtime instances were unable to send dyno status updates, and were flagged as “lost”. Many of these “lost” instances were still hosting dynos, but we lost all visibility into these and into any dynos which were stopped or crashed. We realized the issue but, by this point, some of the runtime instances were starting to use the new credentials.

We rolled two of the Redis servers back to the old credentials, so that the dyno manager could communicate with all runtimes in the fleet. Unfortunately, other bugs were uncovered during the incident: an incorrectly trapped exception in the runtime agent software caused failures when connecting to Redis servers with incorrect credentials, which delayed recovery. At the same time, PX dynos running in a special dedicated fleet didn’t receive the change in credentials due to a configuration issue, and remained impacted for longer than expected.

What will we do to mitigate problems like this in the future?

We’ve fixed the operational documentation that lead to this incident and are conducting a full review of our operational documentation across the board. More broadly, we’re focussing on a stronger risk assessment of all of our procedures, including maintenance scheduling. We’re also improving our rollback plans, so that we can make changes more safely and so we can respond to issues more quickly in the future.

We are reviewing our internal processes to ensure that communication between groups is more effective, so that we can better inform our customers when situations occur.

Further, the software bugs uncovered during the incident were fixed and automated tests were included to prevent regression.