Latency issues

Incident Report for Buildkite

Resolved

We have completed the provisioning of additional capacity mentioned in our last update, and error rates and response times have returned to normal.

This incident is now resolved.
Posted Mar 04, 2026 - 05:24 UTC

Update

We continue to observe high latency on isolated infrastructure serving Agent API endpoints for a subset of customers. We are provisioning additional capacity to address this latency, and have informed impacted customers.
Posted Mar 04, 2026 - 03:29 UTC

Update

We've seen a small number of unrelated issues, each affecting a subset of customers. Most impact is resolved, but we are continuing to monitor impact for a small number of remaining customers. We are in touch with those customers directly.
Posted Mar 04, 2026 - 01:06 UTC

Monitoring

We've made some changes to address the issue and are seeing signs of recovery. We continue to monitor the situation.
Posted Mar 04, 2026 - 00:11 UTC

Update

We continue to experience high latency on some services. We're continuing to identify root causes.
Posted Mar 03, 2026 - 23:21 UTC

Update

We're still experiencing latency issues for agent api and job dispatch. We continue to investigate and identify the root cause.
Posted Mar 03, 2026 - 22:41 UTC

Investigating

We're seeing elevated job dispatch latency and Agent API latency across multiple shards. We're investigating.
Posted Mar 03, 2026 - 21:51 UTC
This incident affected: Agent API and Job Queue.