The past few weeks we’ve been preparing to roll out Content Security Policy (CSP) header support, which adds further protection from a number of XSS attacks. In the process of rolling out some diagnostic changes for CSP we made a change that caused the Buildkite’s Javascript to stop loading, rendering the Buildkite web interface unusable for the majority of browsers.

What change was deployed?

Weeks ago we deployed a new CSP policy in report-only mode, and because our frontend assets are served from a different host (buildkiteassets.com) the CSP reports didn't contain full stack traces. For useful reports we needed to add the CORS crossorigin="anonymous" to all our script tags.

After reviewing the asset server’s configuration (the AWS S3 bucket CORS policy), reviewing the code changes, and running our integration and unit tests, we then deployed crossorigin="anonymous" attribute support on Oct 26 at 15:01PM AEST.

For some users (mostly Safari) there was no immediate problem, but shortly afterwards we had internal reports of the buildkite.com web UI being completely blank in other browsers, followed by the same reports from customers.

We immediately reverted the changes and removed the crossorigin attribute, and 5 minutes later the web interface was fixed and available again.

What went wrong?

Adding a crossorigin="anonymous" tag attribute causes the browser to send a CORS request. This means that it sends an Origin request header with its request, and requires that the response contains a matching Access-Control-Allow-Host header.

Unfortunately our responses were not being served with the correct Access-Control-Allow-Host header and were blocked from loading. We serve our buildkiteassets.com from an S3 bucket via Cloudflare. And though we'd verified the CORS policy configuration in the AWS S3 console, we hadn't performed a manual check of buildkiteassets.com HTTP responses. And due to a quirk in S3’s CORS header implementation, we had misunderstood how our CORS policy was going to be applied.

Stale caches

Later that day we had fixed the S3 configuration, and buildkiteassets.com responses were being served with the correct CORS headers.

We rolled out crossorigin="anonymous" support again in a new deployment, internally tested production and found no problems. We also waited to see if any users reported problems. No problems could be found, and we considered the problem fixed.

A few hours later we had reports of people getting a blank page on Buildkite, but fixable by force-refreshing the page. We immediately rolled back the crossorigin attribute change yet again, and investigated the issue.

We discovered that because we use immutable script URLs that don't contain a Vary: Origin header, browsers would use old cached responses that didn't contain the required CORS headers (see this chromium bug comment for details). Our existing asset URLs were unusable.

Even though this problem was fixable by users (by doing a browser refresh) the time it took to detect and respond to this issue was unacceptably long.

Moving our CSP support forward

The only way for us to roll out crossorigin="anonymous" support without issues would have been to:

1. Manually verify that all assets are served with a Vary: Origin header, or with the correct Access-Control-Allow-Origin headers.
2. Bump the URLs for all the assets, to ensure that no browsers are using a cached asset that is missing the CORS response headers
3. And only once those two things are done, deploy the change that adds the crossorigin="anonymous" to the script tags

We're already in the process of rolling out a new asset host endpoint (using AWS Cloudfront and Lambda Edge) which gives us complete control of the outgoing HTTP headers, and adds the correct CORS headers.

Improvements we’re investigating

• Perform deployments of this nature on a much smaller surface area, such as a small sub-section of the application or on a canary machine
• Add automated post-deployment checks in a full browser environment with cached and uncached assets
• Perform more rigorous verification and testing of assumed S3 behaviours
• Investigate methods for detecting production JS loading failures in browsers
• Enforce CSP policies in our test suite using the SecurityPolicyViolationEvent
• Removing the asset host altogether (which would also minimise the additional DNS latency and HTTP2 socket)