Increased request latency for API

Incident Report for Buildkite

Resolved

Our performance is back to normal and this incident has been resolved. We'll continue working to prevent it from occurring again.
Posted Apr 25, 2022 - 21:45 UTC

Monitoring

We've identified low capacity on API endpoints and we applied mitigation; we are monitoring its results. We are seeing performance return to normal.
Posted Apr 25, 2022 - 20:48 UTC

Identified

The issue has been identified, and we are working on mitigations.
Web UI is currently operational, there may be some particular dashboards timeouts (query all pipelines for example)
Posted Apr 25, 2022 - 20:29 UTC

Update

We are continuing to investigate this issue.
Posted Apr 25, 2022 - 20:04 UTC

Investigating

We are currently investigating increased latency in our APIs and web UI.
Posted Apr 25, 2022 - 19:26 UTC
This incident affected: Web, Agent API, and REST API.