Overview

On Saturday the 25th of November, some of Buildkite's background workers stopped being scheduled correctly. This prevented build jobs with a timeout_in_minutes from accurately timing out.

The incident was not caught by our monitoring systems, and ultimately brought to our attention by a customer in the evening of Tuesday the 28th of November. The following morning we saw these messages and began an investigation.

As of Wednesday the 30th of November, we have fixed the underlying scheduling issue and made workers more resilient to these kinds of changes, and are working to ensure we have correct monitoring of these subsystems.

Apologies to anyone whose builds were affected. 🙇‍♂️🙇‍♀️

Root Cause

The background workers responsible for timing out jobs stopped during minor maintenance on Saturday the 25th of November which switched our Sidekiq workers to a new Redis database. They should have resumed work immediately afterwards, but did not.

We traced this failure to these workers only scheduling subsequent runs of the workers themselves, meaning they needed manual initiation on the new Sidekiq instance.

Changes We’ve Made

• We immediately restarted the affected workers, and all affected jobs properly timed out shortly after.
• Switched to a more resilient scheduling mechanism (sidekiq-cron, with a rate limiting mechanism) for such workers.
• Added granular monitoring of our scheduled background workers using Cronitor.

Ongoing Improvements

• Improving the time between customer report and investigation, even when all metrics and monitoring are reporting as healthy.

Please contact us if you have any questions or comments at hello@buildkite.com