BrowserStack Security Incident Report — November 2014

Published: November 10, 2014 Signed: Ritesh and Nakul, Founders, BrowserStack

Original Incident Statement

On November 9, 2014 at 23:30 GMT, an attacker gained unauthorized access to a portion of BrowserStack’s registered user email addresses and used them to send a misleading email to less than 1% of users (an estimated 5,000 people), falsely claiming BrowserStack was shutting down. Once the company became aware, it temporarily took the service offline to carefully audit every component, causing several hours of downtime for users.

What happened?

BrowserStack’s application runs on a large AWS deployment of thousands of servers. One machine — an inactive prototype server running since before 2012 — had not been retired or patched and was compromised via the Shellshock vulnerability.

The old prototype machine still held BrowserStack’s AWS API access key and secret key. The attacker used these to create a new IAM user and key pair, spin up an instance inside the AWS account, and mount a backup disk belonging to a production component service. That backup contained a config file with the database password. The attacker then whitelisted his own IP on the database’s security group (AWS firewall) and began copying a table containing partial user data — email addresses, hashed passwords, and last-tested URLs.

The copy operation locked the database table, which triggered a monitoring alert. BrowserStack’s team spotted the unrecognized IP in the logs and blocked it immediately, cutting the exfiltration short. Using the partial data and BrowserStack’s Amazon SES credentials, the attacker then sent the misleading email described above to a small subset of users.

What was the extent of the damage?

Database logs confirmed only partial data was copied and no user test history was affected. Credit card details were not compromised — BrowserStack stores only the last four digits of card numbers, with payment processing handled entirely by a third-party partner. All user passwords were salted and hashed with bcrypt, which the company described as effectively uncrackable, though it still recommended users change their passwords as a precaution.

AWS CloudTrail logs verified no other services or machines were touched and no AMIs or other data stores were copied. Production web server logs showed additional Shellshock exploit attempts around the same time, but those failed because the active production servers were already patched.

Points in the email

The attacker’s email quoted three claims from BrowserStack’s security documentation, which the company addressed directly:

• Tamper-proof restoration — confirmed accurate. Every test machine is snapshotted, and restored to that clean snapshot after each session, wiping out anything a prior user installed.
• Network security — confirmed. Every machine sits behind an OS firewall plus EC2 security groups as hardware-firewall equivalents, with brute-force throttling in place.
• Session privacy — confirmed. Each VM is allocated to one user at a time with a randomly generated, session-specific VNC password, so not even BrowserStack administrators can see a user’s session.

BrowserStack also clarified that the plaintext-looking credentials referenced in the attacker’s email were leftovers from the company’s early prototyping phase years earlier (one was an old, weakly-hashed VNC password regenerated every session; the other was a system-user password from before the company switched to key-based SSH authentication with root login disabled). It confirmed VNC still ran on the same port but required a per-session password, which it did not consider a vulnerability.

Where did we go wrong?

BrowserStack acknowledged two core mistakes: every server — active or not — should have received security patches, including the Shellshock fix; and any inactive server should have been stopped and stripped of AWS keys rather than left running with live credentials. It also acknowledged that communicating with users incrementally as the investigation progressed would have been better than waiting to publish one complete account.

Security measures taken to mitigate and prevent further incidents

• Revoked all existing AWS keys and passwords and issued new ones immediately.
• Audited SSH logs, web server logs, and AWS CloudTrail records to confirm the scope of the breach.
• Began migrating all backups to encrypted storage and removing unencrypted copies.
• Added new checks and alerts on specific AWS actions.
• Regenerated all VM snapshots, replacing the existing ones.
• Began evaluating VPC/VPN options to further isolate infrastructure.
• Commissioned an external, independent security audit.
• Traced the attacker’s IP and prepared to file a report with authorities.

The founders closed by apologizing for the inconvenience and noting they had identified the attacker’s IP and intended to pursue an official complaint with authorities.