Overview

We had a very brief (7 minutes) outage of our job logs sub-system, which had cascading impact on basic usage of the platform. No data was lost as our agents are designed to retry in the face of this sort of failure, but during the outage API requests would fail and customers would be greeted with blank job logs in their builds.

We also neglected to correctly mark the subsystems on our buildkitestatus.com as down, which means the outage didn't affect our recorded uptime. We can't do this retroactively, but we will be putting measures in place to avoid this in future.

Sorry about the interruption! Many thousands of developers count on Buildkite being available, so we take even short outages like this really seriously. 🤕 🙇‍♂️ 🙇‍♀️

Root Cause

Our job logs are cached in a PostgreSQL database prior to being written out to S3, and this database became full after a spike in job logs. We subsequently discovered that the archival mechanism was skipping some data, causing a slow build up of logs. This database has historically been a very short term store, so it lacked alerting around low disk space such that we could manually intervene.

Changes We've Made

• Cloudwatch alerts on disk space at 50% so we have time to fix things before it gets worse
• Improve logic for archival of logs from the Job Chunks DB to S3
• Migrated database to RDS Multi-AZ for faster configuration updates and updated the database version

Ongoing Improvements

• Training staff around how to use StatusPage's components in an outage
• Investigating better integration with PagerDuty and StatusPage to largely automate marking components as degraded

Please contact us if you have any questions or comments at hello@buildkite.com