Buildkite service disruption

Incident Report for Buildkite

Resolved

For the last week we have been internally testing our agent traffic running on our new EKS infrastructure successfully.
At 23:01 11 August 2026 UTC we started routing < 3.97.0 saw an error "failed to decode JSON response: invalid character '
We reverted the routing to EKS infrastructure at 03:25 UTC after being alerted to the issue.
Posted Aug 12, 2026 - 04:08 UTC

Monitoring

We have reverted the change which introduced the elevated error rate. We believe the issue to be resolved but will continue to monitor.
Posted Aug 12, 2026 - 03:27 UTC

Update

For a subset of customers, <1% of their Agent API traffic is experiencing issues. We are rolling back the change that caused this issue while we investigate it further.
Posted Aug 12, 2026 - 03:22 UTC

Identified

We're seeing increased latency and error rates for a subset of our customers. We're currently addressing the cause and will provide status updates as they become available.
Posted Aug 12, 2026 - 02:59 UTC

Investigating

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Posted Aug 12, 2026 - 02:44 UTC
This incident affected: Agent API.