High error rates and timeouts in the Agent and Rest APIs

Incident Report for Buildkite

Resolved

This incident has been resolved.
Posted Jul 13, 2024 - 22:52 UTC

Update

We are continuing to monitor for any further issues.
Posted Jul 13, 2024 - 22:42 UTC

Update

We are continuing to monitor for any further issues.
Posted Jul 13, 2024 - 22:25 UTC

Monitoring

We have identified the source of high load and put service protections in place to reduce the impact. The Agent and Rest APIs are now performing nominally. We continue to monitor service performance.
Posted Jul 13, 2024 - 22:10 UTC

Update

We are experiencing degraded database performance which is causing high latency and timeouts on the Agent and Rest APIs. We continue to investigate the root cause. We will provide another update in 30 minutes.
Posted Jul 13, 2024 - 21:36 UTC

Update

We are experiencing a high concurrence of Builds which is causing increased latency and timeouts on the Agent and Rest APIs. We continue to investigate the root cause. We will provide another update in 15 minutes.
Posted Jul 13, 2024 - 21:17 UTC

Investigating

We've spotted an issue with our Agent and Rest APIs. We're currently investigating the issue, and will provide an update soon.
Posted Jul 13, 2024 - 20:39 UTC
This incident affected: Agent API and REST API.