Identified - Since Wednesday 22nd April, customers have experienced intermittent periods of increased latency and request timeouts. These periods are most common during North American business hours, and are most commonly seen as HTTP 502 errors in builds that upload or download artifacts.

Database resource contention is at the root of the problem, with two overlapping issues:

1. We use a per-build lock around many agent operations (like pipeline uploads, job accepting, and job starting). This has ensured correct behavior, but as we've grown and our customers have built novel workloads, the lock contention has become a bottleneck that can't be solved by throwing more CPUs at the problem (we tried)
2. The locking happens in a database table with a high update rate and the indexes have become large enough to impact general query performance

So far we've upgraded the size of our database, worked with a few customers to adjust their workloads to a form that uses fewer resources, and profiled locking hotspots. Our next step is to introduce finer grained locking around some operations and adjust our database schema to ensure querying data related to active builds is faster.

We'll post updates here as we make progress, and share a post-mortem when the issue is resolved.

Our agent has robust back-off and retry behavior when communicating with our APIs, which means even in the face of API timeouts or errors, these database issues generally manifest as slightly delayed builds. However we've identified one interaction that wasn't retrying, resulting in build errors on API timeout. We've released a new version of the agent (https://github.com/buildkite/agent/releases/tag/v3.21.0) with improved retry behavior for artifact uploads and downloads. Customers may be able to mitigate some of the impact by upgrading.
May 5, 14:12 UTC