On February 28 from 17:37 UTC to March 1 00:18 UTC, Heroku customers experienced significant disruptions in service, outlined in detail below. Heroku users rely on the availability of our services to run their apps and, in many cases, their businesses. We take both availability and uptime very seriously. We sincerely apologize for the impact this outage had on our customers and users.

A great deal of thought and care goes into which providers and technologies we use to build Heroku. Any instability or unavailability due to issues with those providers or technologies is a consequence of our choices.

Many of the problems during this incident stemmed from the Amazon S3 outage on the 28th. Below we provide details on which of our services were affected and why they were affected. We also outline the steps we’re taking to mitigate the impact of future outages of a similar nature.

What was affected, and how will we improve?

Heroku’s service was disrupted in many ways. After every incident that disrupts our service, we conduct a thorough retrospective analysis to fully identify customer impact, causes, and ways to reduce or avoid such impact in the future. In each of the following sections, we’ll describe the impact to a service, explain why it happened, and share the improvement plans we identified.

Dynos and Releases

Amazon S3 (“Simple Storage Service”) experienced a severe outage. We use S3 to store slugs for all applications and also releases for applications in Private Spaces.

Every time we start a dyno, our system fetches the current slug from S3. These fetches experienced near total failure resulting in an inability to start new dynos. Existing, running dynos continued to run unimpeded, and Heroku Routing continued to route requests to them.

Failure to start new dynos impacted apps in several ways:

• Restarts: When an application was restarted, the old dynos were terminated but the new ones could not start. Even with preboot enabled, the old dynos are terminated after 3 minutes. If all dynos in an application were affected, requests to that application were met with Error H10.
• Cycling: Dynos that were automatically (“cycled”) were terminated and did not restart.
• Sleeping: Free apps that went to sleep were unable to wake up.
• One-off dynos: One-off dynos could not start, which affected Heroku Scheduler, release phase commands, and manually created heroku run processes.

With the exception of one-off dynos, affected dynos restarted in the time between February 28 21:54 UTC and March 1 00:28 UTC.

Improvement Plan

The best way we can improve customer impact from an impairment in our ability to start dynos is to prevent them from being stopped unnecessarily. Our long-standing practice during incidents of this sort is to disable dyno cycling and sleeping. In our post-incident analysis, we discovered that our ability to take these actions was hampered by the S3 outage. We have identified several ways to improve our tools and response procedures in order to speed our ability to take these actions in the future.

We are also looking into ways to improve our resilience to S3 outages and allow dynos to continue to start.

API Maintenance

The Platform API provides a method for control over all aspects of applications. It is used by the Heroku CLI and Dashboard to provide their services.

We disabled the Platform API to prevent customers from performing actions that we knew would result in their application becoming unavailable and returning H10 errors. This also had the side-effect of preventing all CLI and Dashboard operations, including potentially safe operations such as viewing running dynos, configuration variables, or authenticating. This impact applied to all regions of both the Common Runtime and Private Spaces.

Improvement Plan

As in the previous section, this incident exposed problems in the tools and procedures used to disable the Platform API. We are in the process of rectifying these deficiencies so that we can disable the Platform API more quickly in similar incidents.

We also identified substantial room for improvement in our procedures and documentation around making the decision to disable the Platform API. We had two options:

1. Completely disable all API actions for all Runtimes in all Regions.
2. Disable a smaller subset of API actions that are specifically related to modifying applications. Actions can be disabled for some runtimes and regions independently.

We started with option 1 for the first several hours of the incident and switched to option 2 toward the end. Option 1 is a drastic action that completely prevents our customers’ ability even to retrieve information about their running applications. This hampered some customers’ ability to enact their Disaster Recovery (DR) plans.

We try only to use option 1 when signs indicate that even partial access to our API would cause damage. In our retrospective analysis, we determined that our incident responders had insufficient and conflicting information that led them to believe that option 1 was required when it was not. We have identified several ways to improve our procedures in order to make this decision process less difficult and more effective.

We also found room for improvement in option 2. Our tools did not make it possible to limit the disabled regions and runtimes to precisely those impacted in this incident. To remedy this, we will seek to add more granularity to this option.

Heroku Data

A small fraction (about 0.03%) of Heroku Data add-ons (Postgres and Redis) became unavailable during the incident. New Heroku Data add-ons requested during the incident were not provisioned until after it was resolved.

For the duration of the S3 impairment, transaction data in Heroku Postgres databases in the US region were not automatically archived. Heroku databases have a fixed amount of space available to store WAL files awaiting archiving to S3. In about 0.0013% of Heroku Postgres databases, the level of write traffic caused WAL files to exhaust the available storage space. This rare condition caused these databases to crash and require manual intervention from our engineers. Once S3 became available, our engineers immediately undertook this maintenance to bring these databases online. Although this affected a small number of databases overall, and no data was lost, the time to recover was significant and we apologize to those customers affected.

Normally, Heroku Data add-ons at the Premium level are Highly Available (HA). In the event that a host fails, a secondary host is ready to take over and the application’s configuration is automatically updated to point to it. While the Platform API was disabled, HA failover was impaired because the configuration changes could not be made. Engineers monitored this situation and rectified any inconsistencies after the Platform API was re-enabled.

Improvement Plan

We seek to automate recovery of crashed Heroku Data add-ons whenever possible, both to restore them as quickly as possible and to eliminate the possibility of human error. The case where a Heroku Postgres fills its available WAL storage space is extremely rare, and as such it is one of very few recovery actions that we have not yet automated. This incident gave us valuable information on this kind of crash that we can use as we automate recoveries of this sort.

Additionally, we are investigating the possibility that our system could attempt to update application configuration in the event of an HA data add-on failure even when the Platform API is disabled.

Heroku Connect

Connect experienced an interruption to polling-based synchronization. Mappings configured with Streaming Mode continued to synchronize data from Salesforce, and writes to Salesforce were unaffected.

Improvement Plan

As with many of our services, we run Connect on the Heroku Platform. We expect other improvements within the platform will aid in recovery from similar failures in the future. We will also investigate ways to make the service more resilient to the inability to launch new dynos.

EU Region of the Common Runtime

Applications in EU were impacted as well, though to a significantly lesser degree than in the US. Applications with no releases since August 2016 may have their slugs stored in the US. This resulted in impact similar to that seen by applications in the US region.

Extended Impact After S3 Outage Resolution

In situations such as this, we strive to restore Heroku to normal operation as quickly as possible. However, in this incident, impact to Heroku extended well beyond the time of resolution of the S3 incident stated by Amazon. AWS services other than S3 continued to be impaired following S3’s recovery, and this hampered our ability to gain access to additional AWS resources in order to recover our systems.

Additionally, 9.5% of applications in Private Spaces had extended impairment. A bug in one of our components, triggered by the S3 outage, caused some dynos to enter a faulty state. Requests that reached these dynos returned an H21 error until our engineers manually intervened.

Improvement Plan

We work closely on a regular basis with Amazon to troubleshoot problems that impair the reliability of our service. This incident provides us with an excellent opportunity to discuss disaster recovery scenarios with them and use the resulting information to improve our incident response procedures.

Shortly after the incident, our engineers found the bug that caused H21 errors and commenced work on a fix. We sincerely apologize for the impact that this defect had on our customers’ applications, and we continue to look for ways to catch this kind of problem before it causes customer impact.

Availability Metrics

Our Status site shows an Availability percentage for the US and EU regions of the Common Runtime. Many customer applications experienced significant downtime, and the individual experience for those customers did not align with the numbers currently displayed. We’d like to share some details on why this is the case and what we plan to do about it.

Our availability calculation methods are detailed in our Dev Center article on Heroku Status. Due both to the nature of our canary applications and random chance, these applications were not impacted by this incident. Conversely, many customer applications experienced crashes, and crashed dynos were not restarted, resulting in extended downtime for those applications.

During the incident, Heroku Routing experienced very little impact. Since our canary applications did not crash, our availability metrics for this time period primarily reflect just the availability of Heroku Routing itself.

Improvement Plan

Transparency in our availability reporting is incredibly important to us. In light of this incident, we are carefully reviewing our availability calculation methods in an effort to better reflect the full range of customer experience, especially during outages of this nature.

Due to the sensitive nature of these metrics, we have made the difficult decision to leave the current numbers unchanged rather than attempting to modify them for this incident. We feel that the likelihood of releasing erroneous or misleading information is too high if we make such a change in haste.

Moving Forward

We take all impairment to the normal operation of our service very seriously and especially incidents of this magnitude. This incident has highlighted multiple areas for improvement in our systems and procedures in the short and long term. In this way, we hope to significantly lessen the impact of such incidents in the future.