We have seen full recovery for customers since 20:28 UTC. We experienced an autoscaling feedback loop which increased the number of connections to our redis cluster above its ability to respond. This had widespread impact for all of our customers with Web UI, Agent API, REST API and job queue impact between 18:43-19:21 UTC, and again between 20:02-20:28 UTC. A full post incident review will be available later this week.
Posted Sep 10, 2026 - 01:54 UTC
Monitoring
We are seeing improvements across the affected services, and are seeing services return to normal functionality. We are continuing to monitor and are determining the root cause.
Posted Sep 09, 2026 - 20:53 UTC
Update
We are continuing to investigate elevated error rates across multiple services, and are working to determine the cause.
Posted Sep 09, 2026 - 20:28 UTC
Update
We are continuing to investigate this issue. We are seeing impact on the Agent API which will affect job scheduling, artifact uploads, and an increase in 5xx responses from the Agent API endpoints.
Posted Sep 09, 2026 - 19:41 UTC
Investigating
We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Posted Sep 09, 2026 - 19:07 UTC
This incident affected: REST API (REST API) and Web, Agent API, Job Queue.