Beginning on August 11th around 13:10 UTC until 14:04 UTC, apps in our EU region experienced high request latency and possible connection errors. We sincerely apologize for the problems this caused.
Here is some additional detail about what happened and steps we are taking to mitigate future outages of a similar nature.
Who was affected?
EU apps experienced intermittent request latency and connection errors, up to multiple seconds per request, for about an hour.
What Happened?
Our routing layer had a latent bug which had existed for the lifetime of the codebase, resulting in an artificial shortage in the total number of active connections the routing layer could support. A specific (non-abusive) customer workload caused the routing layer in the EU region to run up against this limit for the first time. Because this was an artificial shortage, the routing nodes were largely idle. This presented a red herring which slowed our diagnosis and response.
What will we do to mitigate problems like this in the future?
The bug in question was previously unknown and affects both the US and EU regions of the Common Runtime. Private Spaces are not affected.
We have identified the fix and have rolled it out to both regions. We’ve discovered that logplex is also vulnerable to this issue and we will deploy a proactive fix for it. As the issue could impact similar code in other languages (notably Go), we are in the process of checking other parts of our infrastructure to determine if they’re vulnerable to this issue.
The bug involved an unexpected pitfall in a certain pattern of Linux system calls involved in establishing an outgoing TCP connection. The debugging process was complex and intriguing, and we believe the broader system operations community may find it useful. To that end, we will write up our investigation process and post it to the Heroku Engineering Blog.