On January 10, 2018, beginning at 18:48 UTC, applications experienced a full or partial loss of log data delivery to drains. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
All applications using log drains or log tailing functionality with the exception of customers using Private Space Logging were affected.
What happened?
On January 3, 2018, AWS began patching all EC2 infrastructure in order to mitigate the Spectre/Meltdown vulnerabilities. These patches were known to cause slightly degraded performance for some workloads. As a result of this patch, our logging service began to exhibit elevated CPU load.
This elevated CPU load caused the logging service to periodically reach continuous 100% CPU saturation. As CPU resources were exhausted, the instances were unable to keep up with the workload and began to use additional memory to store uncompleted work, which led to memory exhaustion. When this occurred, the Erlang VM automatically began to kill processes, leading to service degradation and, at times, complete failure while we executed recovery operations.
The service degradation reached the threshold of our alerting on January 10 at 18:48 UTC. On-call engineers responded and began to diagnose the issue.
Over the course of 48 hours, we made several attempts to restore service. Our initial response was to horizontally scale the service, but we reached our maximum horizontal scaling limit for this service given the implementation at the time. We also updated Linux kernels, changed DNS resolution settings, turned off high-load non-critical services, and modified connection limits. Unfortunately, these modifications failed to produce significant results.
Upon discovering the change in performance correlated to the rollout of the AWS patches, we engaged with our AWS team while continuing to work on our own remediation. Service was restored following additional AWS patches that restored performance to previous levels.
What we will do to mitigate problems like this in the future?
We have put new controls in place to isolate and minimize future impact. Work is underway to increase our ability to scale our cluster size when faced with this class of resource contention in the future. Long-term work is planned to enable us to utilize larger instance sizes for this service.