At 2023-09-18 18:45 UTC, AWS announced “Network connectivity issues” impacting Elastic Container Service, Elastic Compute Cloud and Elasticache (among others) in Buildkite’s primary AWS region.

At this exact time, one of our Sidekiq services—then serving as the elected leader for enqueueing the periodic jobs used to run scheduled builds—experienced a matching loss of network connectivity, manifesting as errors connecting to AWS and other external services.

From this point, this service no longer enqueued these periodic jobs, while also consuming 200% of its container’s CPU.

Sidekiq leadership renewal should happen every 20 seconds, and because no other Sidekiq process took leadership, we theorise that this one process was able to renew its leadership but was otherwise in a state preventing its ordinary operation.

This meant that scheduled builds did not run over this period.

At 2023-09-18 21:56 UTC, a Buildkite deploy started new Sidekiq processes, resulting in a new leader election and a new Sidekiq process taking the role of enqueuing the periodic jobs, and scheduled builds successfully resumed.

This is the first time we’ve seen a Sidekiq process retain its leadership while otherwise ceasing to function normally.

To ensure we can respond better to such an occurrence in future, we’re establishing a dedicated monitor for the periodic job used to run scheduled builds.