Potential Platform Issues (2017-02-02 18:38 UTC)
Engineers are investigating potential issues with the Heroku platform.
Update (2017-02-02 18:45 UTC)
We’re currently investigating an issue involving permission errors encountered when attempting to write to the filesystem on some dynos.
Update (2017-02-02 18:51 UTC)
We’ve identified the cause of the issue and are working on getting this fixed on all affected servers.
Update (2017-02-02 19:23 UTC)
The issue has been solved on all servers in the EU region. The fix will only apply to newly booted dynos though. If you’re being impacted by this incident in EU, restarting your dynos via the CLI with heroku ps:restart will get things working again. The fix is being rolled out for the US now.
Update (2017-02-02 19:43 UTC)
The issue has been solved on servers in all regions now. The fix will only apply to newly booted dynos though. If you’re being impacted by this incident still, restarting your dynos via the CLI with heroku ps:restart will get things working again.
Resolved (2017-02-02 20:18 UTC)
This incident is now resolved. All new dynos that boot are no longer affected by this. If you’re still seeing issues, please restart your affected dynos via the CLI with heroku ps:restart.
Follow-up (2017-02-14 19:31 UTC)
Beginning on February 2nd, between 14:29:38 UTC and 20:18:24 UTC, our customers experienced write permission failures to the /dev area of the file system, in particular /dev/null. We accept full responsibility for any downtime arising from this issue and apologize for negative effects our customers experienced.
Here is some additional detail about what happened and steps we are taking to mitigate future outages of a similar nature.
Who was affected?
Customers in the EU and the US with applications or applications using buildpacks that attempt to write to any files located under /dev, such as /dev/null. The write permission failures likely caused applications to crash (H10 errors) or have difficulty booting (R10).
The number of affected customers would have slowly increased over the extended period of the incident.
What Happened?
An upstream patch update to apt caused our customized source policy to be ignored. This meant that pinned packages in our custom repository were skipped and default upstream packages were installed. The upstream source contained versions of packages that caused the changes to the underlying filesystems for our customers.
What did we do to prevent wider impact?
As soon as we detected the problem we immediately paused the creation of new instances in our slack pools. This prevented the impact from spreading to new instances.
We identified the affected servers and began the process of rolling back packages. Once the packages were rolled back to versions from our apt sources new dyno boots could succeed.
What will we do to mitigate problems like this in the future?
We have already updated our apt policy file to ensure that both newer versions and older version will pull our customized packages from our sources. This fixes the immediate problem. Furthermore, we have a plan moving forward to improve our validation processes for new instances.