Beginning on October 21 around 11:00 UTC until 13:15 UTC, and again from 16:00 UTC until 17:00 UTC, the Heroku platform, website, and status site experienced a partial outage. The incident affected deploys, scaling, our API, and the Dashboard. Running apps were unaffected, but some traffic was unable to reach them due to a DNS outage, rendering some apps effectively down.
Heroku users rely on the availability of our services to run their apps and, in many cases, their businesses. We take both availability and uptime very seriously. We sincerely apologize for the impact these outages had on our customers and users.
A great deal of thought and care goes into which providers and technologies we use to build Heroku. Any instability or unavailability due to issues with those providers or technologies are a consequence of our choices.
The problems during this incident were caused by a Distributed Denial of Service (DDoS) attack against one of our DNS providers, Dyn. Below we provide details on which services were affected and why they were affected. We also outline the steps we’re taking to mitigate the impact of future outages of a similar nature.
What was affected, how was it affected, and why?
Various DNS domains are involved in providing the Heroku service. These include heroku.com and herokussl.com, among several others. These domains in particular were hosted using Dyn’s DNS servers, and they were the primary source of impact to our services.
All impact was varied in its geographical scope and duration. Due to the way Dyn’s DNS service is set up, your request may be routed to a Dyn datacenter close to you. The attack targeted Dyn’s worldwide network of datacenters, but at any given time, only some of them were down. This means that whether you saw impact depended in some cases on the time and where you were located.
Heroku.com, status.heroku.com, and related properties
During the incident, our web properties had episodes of unavailability. This meant that our Main Site, our Support Site and our Status Site were all impacted. Customers would have been unable to manage their apps through the Dashboard, file support tickets, or see our Status Site postings about the incident.
Heroku CLI and API
Access to the Heroku CLI and API was also severely impaired. In some cases, the Dyn datacenter that your DNS request was routed to was available and your API request made it to our servers. However, our API servers also had a dependency on DNS resolution in the heroku.com domain, so your API request may still have failed, resulting in an HTTP 500-series response.
This meant that operations such as scaling, deploying, restarting, and examining running applications were severely impaired.
Logging
Logging was impaired, again due to an internal dependency on DNS resolution in impacted domains. Apps received L13 errors indicating that log messages were dropped.
Routing of Application Requests
Requests to some apps were unable to reach our platform due to DNS lookup failures. Overall, we saw a 20% decrease in requests to our platform.
Requests to SSL Endpoints were impacted, because domains hosted on SSL Endpoints have DNS CNAME records with targets in the impacted herokussl.com domain. Requests to any customer-owned app domain that itself was hosted using Dyn were also impacted.
Requests to domains hosted on the new Heroku SSL offering were not impacted. These domains have DNS CNAME records with targets in the unimpacted herokudns.com domain.
Requests to domains for apps in Private Spaces were similarly not impacted. These domains have DNS CNAME records with targets in the unimpacted herokuspace.com domain.
Finally, HTTP and HTTPS requests made to herokuapp.com domains such as your-app.herokuapp.com were not impacted.
What did Heroku do to mitigate impact during the incident?
Our incident response was severely hampered by the standard TTL (“Time To Live”) of DNS server records on .com domains, which is set to 48 hours and cannot be changed. This meant that any attempt to switch our domains to use a different DNS provider would take up to 48 hours to come into effect for all customers. For this reason, we prioritized other mitigation efforts more highly than switching providers.
Increased TTLs
We increased the TTLs for various DNS records under heroku.com to one day. In cases where an ISP’s caching DNS resolver successfully resolved a given DNS record, it was then able to cache the result for longer, effectively mitigating impact for the subset of users behind that DNS resolver.
Status Site
We feel that communication with our customers, especially during an incident, is of critical importance. The unavailability of our Status Site, especially when coupled with the unavailability of our Support Page and the @herokustatus twitter account (due to a Twitter outage) meant that our customers were unable to get timely information about the outage. We sincerely apologize for this failure.
As a stopgap measure during the incident, we created a temporary secondary URL for the status site. See below for more information on how we will improve communication in the future.
Workarounds
We recommended workarounds to help customers deal with the impact of the outage. Customers using SSL Endpoint could migrate their app to the unimpacted Heroku SSL.
We also recommended that customers modify their /etc/hosts file temporarily to avoid the need for DNS lookups. We urge customers to revert /etc/hosts changes to avoid risk of future impact, as IP addresses are subject to change.
What will we do to prevent problems like this in the future?
This outage exposed a critical weakness in our DNS hosting configuration. We are taking immediate steps to add additional DNS providers. This should allow us to avoid impact in the future, provided that at least one of our DNS providers is operational.
We will also move the Heroku Status page to a new domain that does not share the same DNS providers as our other domains. We will publicize the new URL once it is available.
Additionally, we will investigate reducing or removing the dependence of our internal systems on external DNS servers. This would improve the ability of services like the API to respond to requests during a similar outage.