On September 17 between 5:15 UTC and 7:30 UTC, a subset of our customers experienced failed requests and increased request latencies to their EU Common Runtime applications. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
55% of the EU Common Runtime applications were unable to receive requests or would have experienced increased request latencies during this time. However, only a fraction of those impacted applications were receiving traffic at the time of the incident, so unavailability and degradation would have been felt by 0.6% of Common Runtime EU customers.
What happened?
The EU Common Runtime is fronted by a router service that proxies traffic from clients to customers’ dynos. On August 12, a configuration change was introduced to a resource limit on this router service. Due to natural variations in customer load, this limit was not breached until over a month later, on September 17 at 5:15 UTC. Hitting this resource limit prevented the router service from properly handling clients’ requests to dynos.
At 5:54 UTC on September 17, an on-call engineer received a page for failed requests in the EU Common Runtime. At 6:33 UTC, more router capacity was manually provisioned, but not enough to overcome the resource limit. At 7:13 UTC, even more router capacity was manually provisioned, eventually bringing the total capacity to double what it had been prior to the incident. By 7:30 UTC, the new capacity had come online, and the router was successfully handling requests again, fully restoring service.
What did we do to prevent wider impact?
The misconfigured resource limit on the router instances was identified and corrected in both the US and EU.
What will we do to mitigate problems like this in the future?
A change to remove the code-path that allows for misconfiguring the resource limit in this way is staged for deployment to production. Additionally, the relevant engineering team is exploring a method to monitor this particular resource, so in the future, the on-call engineer can be alerted before the resource is completely exhausted.