Overview

We received a report on Friday 3rd May of a customer losing administrative access to their organization when logging in via SSO with Okta. The problem was resolved by the customer, so we deferred investigation to the following Monday. On Monday 6th May we received additional reports which led us to conclude that we were seeing a widespread issue with a recent set of configuration updates that were published on Okta. We deployed a workaround at 10am on Monday 6th May, which prevented any loss of administrator access, and informed customers that had contacted us.

Root Cause

Our SAML authentication system supports a set of SAML Attributes that can be sent from an Identity Provider along with a SAML Assertion. One of these is “admin”, which can be true or false to indicate that the account is an administrative account.

Okta has a directory of Applications with pre-defined configuration that specifies defaults for use with Buildkite. After in-depth testing of updates to our Okta defaults, we worked with Okta to deploy the new defaults at 5pm Wednesday 1st May (AEST). A side effect of the change, which we hadn’t tested for, was that a new, empty admin attribute is sent if the value isn’t set. Due to an implementation detail in how we handle these attributes, the new admin attribute with an empty string value was evaluated as false, which caused admin logins to drop administrator access.

A significant flow-on effect of the loss of admin privileges, was that some scheduled pipelines owned by the affected admin users, were automatically disabled due to lack of access to their respective pipeline. This disabling of scheduled builds automatically notified account admins, which was one of the ways that we were alerted to the issue.

What We’re Doing

1. We’re putting in place an updated test plan for future Okta configuration rollouts that will include all the SAML attributes we support.

2. We’ve added a category to our status page for Okta-related incidents.

3. We’re going to reach out to all affected users to assist in recovering admin status. If you believe this has affected you, feel free to contact us directly on support@buildkite.com