Between 04:40 and 05:34 UTC on 3 August 2026, Buildkite stopped running recurring background work. Every background job process across the product was affected. The customer-visible impact was concentrated in Pipelines.
During this window, scheduled builds did not start at their scheduled time, jobs waited longer than usual before being dispatched to an agent, in some cases by more than ten minutes, and pipeline uploads and inbound webhook processing were delayed. Job timeouts were not enforced, so a job that should have timed out kept running, and agents that had finished or gone away were not cleaned up promptly.
Scheduled builds were not skipped. Every schedule that came due during the window ran once background scheduling recovered, up to around forty minutes late. A schedule that fires more often than that ran once on recovery rather than once for each interval it missed.
No builds, jobs or queued work were lost. Work that was delayed was processed once the incident was resolved, and jobs that errored were retried automatically. The dashboard, the REST and GraphQL APIs, and agents already running jobs were not affected.
At 04:33 UTC we merged a routine update to Rails, the web framework Buildkite runs on, taken to stay current with a security patch. Alongside Rails, that update pulled connection_pool from 2.5.5 to 3.0.2. The 3.0 release made that library's API keyword-only, which changes how every caller has to invoke it. Our review of the change covered our own code's use of connection_pool and updated it accordingly. It did not cover the other gems in our bundle that call the same library.
One of those is Sidekiq, which runs our background jobs. From 04:40 UTC, as the deploy rolled out, three internal Sidekiq threads inside every worker process failed on startup and stopped. The processes themselves stayed healthy and kept working jobs that were already queued, so nothing crashed and no deploy failed. What stopped was the scheduling of recurring work, which is what triggers scheduled builds, dispatches queued jobs to agents, enforces job timeouts and cleans up finished agents. That scheduling currently runs through a single elected process for the whole background job fleet, so it stopped for every product and every customer at the same moment.
Our monitoring alerted us at 04:48 UTC and we identified the change responsible at 04:59. Our first assessment was that there was no customer impact, because everything we could see was still moving. It took another ten minutes to recognise that recurring work had stopped across the whole product, at which point we published a status page and raised the severity. A fix pinning connection_pool back to 2.5.5 was merged at 05:10 and deployed ahead of the normal merge queue. Scheduling resumed at around 05:19 and the incident was resolved at 05:34.
connection_pool is called both by our own code and by gems we depend on. Our review, and the automated review tooling that flagged the change on the pull request, both looked at how we call it and stopped there.
Nothing in the dependency graph prevented the combination. Sidekiq 7.3.9, the version we run, permits any version of connection_pool from 2.3.0 upwards. Sidekiq 7.3.10 added an upper bound that would have refused 3.0, but we had not adopted it.
The failure only appears when a real Sidekiq process starts up. Our test suite does not exercise that path, and our pre-production check verifies that the application boots rather than that background scheduling is still running.
Because recurring work is scheduled by a single elected process for the whole fleet, one failure removed it for every customer at once rather than for a single group of customers.
We have pinned connection_pool to a version compatible with the Sidekiq release we run. This shipped as part of resolving the incident.
We are upgrading Sidekiq to 7.3.10, which declares an upper bound on connection_pool and so makes this combination impossible to resolve rather than something we have to remember to avoid.
We are changing how recurring work is scheduled so that it is no longer driven by a single process for the whole fleet. Each group of shards will schedule its own recurring work independently. This would not by itself have prevented this incident, because the incompatible library was loaded by every process, but it removes a shared point of failure that can currently stop recurring work for every customer at once. This work is already in progress.
We are updating our internal engineering guidance so that an upgrade moving a shared library to a new major version is reviewed against every consumer of that library, including the other libraries that depend on it, rather than only our own code.