Beginning on June 3rd around 16:48 UTC until 17:06 UTC and again from 17:42 UTC to 20:45 UTC, our US API was in maintenance mode, causing a complete development outage in the US region. In addition, EU customers experienced an outage for builds while the US API was in maintenance mode because that system depends on infrastructure in the US. We accept full responsibility for downtime arising from this issue, and apologize for the significant negative effects our customers experienced.
Here is some additional detail about what happened and steps we are taking to mitigate future outages of a similar nature.
Who was affected?
All US customers were unable to deploy or scale applications for a total of 3 hours and 20 minutes. EU customers were able to scale but unable to build apps while the US API was in maintenance mode due to a cross-region dependency. Single-dyno apps may also have experienced H99s while the API was in maintenance mode if the app tried to unidle.
What Happened?
The Postgres database that’s fundamental to the function of our API and to our ability to collect billing data reached a performance cliff due to table bloat. The database’s main tables and indexes were preserving dead rows that were no longer needed, which increased the resources required to update live rows and respond to queries. Many small factors appear to have contributed to the issue (including a slow disk), but we can confidently say that some of the larger ones are due to the overall design of the system and to this database still being on Postgres 9.0. More specifically, the interactions between hot standby followers, lots of small transactions, and the less-aggressive auto vacuum in 9.0 are implicated in this issue.
What did we do to prevent wider impact?
In order to prevent platform instability, we put the API in maintenance mode while we worked to clean up the table bloat. This was done by renaming bloated tables and indexes, re-creating the tables we needed from live rows, and then dropping bloated indexes. We were able to drop over 99% of the contents of the tables and indexes, leaving a much smaller database.
After rotating the bloated tables out, disk IO fell from 20MB/s to 2MB/s because much less work was required to update the new, leaner tables and indexes. The change in IO immediately solved our performance problems.
What will we do to mitigate problems like this in the future?
We developed a detailed plan for how to upgrade the database to Postgres 9.3, which will give us better tools to monitor and analyze performance, as well as improved interactions between autovacuum and hot standby feedback. This will buy us some time and provide an improved understanding of the performance issues in the current API backend architecture, which we can use to design a replacement.
Another beneficial side effect of the upgrade will be to move the database to an EC2 instance that uses Provisioned IOPS Volumes. This will provide more predictable disk performance, with more overhead for high volume disk IO than our previous disk configuration.