On April 27, 2021, between 13:49 and 17:39 Universal Coordinated Time (UTC), the Salesforce Technology team became aware of an issue that was impacting the ability of customers with multi-factor authentication enabled to log into their Salesforce environments across Commerce Cloud, Marketing Cloud, Pardot, Heroku, and core Salesforce production environments. We sincerely apologize for the negative effects our customers experienced.
Who was affected?
Customers enrolled with multi-factor authentication without active authentication sessions were impacted by this outage. The issue only impacted new login attempts. Customers who were already logged in before the incident were not affected by this issue. Approximately 25% of login attempts completed successfully during the peak time of impact.
What happened?
On 26th April at 15:00 Universal Coordinated Time (UTC) a minor update to the Salesforce Authenticator iOS app was released. On 27th April at 14:00 UTC our Engineering teams were alerted of a surge in traffic around the automated approval functionality in what appeared to be anomalous behavior. This traffic surge resulted in a performance issue on our database instance. Upon further investigation, the engineering team identified that the traffic surge started ramping up immediately after the misbehaving iOS release.
What did we do to prevent wider impact?
To prevent performance issues, we applied a hotfix to temporarily disable the automated location-based authentication feature of the Salesforce Authenticator mobile app. We increased the capacity of the affected resources.
What will we do to mitigate problems like this in the future?
We have made a number of changes across teams to reduce the likelihood of this behavior repeating in the future. We have implemented targeted rate-limiting for the endpoints affected in this incident. We will soon be alerting based on metrics identified during this incident. These alerts will improve our time to detect by approximately 1 hour, alerting subject matter experts before customer impact. Within the coming weeks, we will release a new version of the mobile app and deploy changes to the server.