High error rates

Incident Report for Buildkite

Resolved

Buildkite suffered from high error rates for two minutes on Monday, 18th October 2021 from 4:18 am until 4:20 am UTC. The Buildkite Dashboard and all APIs were affected. Some Buildkite Agent activities may have been interrupted, but default retry behaviours should have mitigated interruption. Webhook ingestion was not affected, although ingested webhooks may not have been processed until the errors subsided.

The errors were caused by a migration which should have been zero downtime except for a small mistake. The migration was being closely supervised, and a revert was quickly applied. The mistake has been fixed, and the errors should not recur.

Posted Oct 18, 2021 - 04:21 UTC