Increased latency and timeouts on REST and GraphQL API

Incident Report for Buildkite

Resolved

We are happy that request latency has dropped to normal levels so consider this issue resolved.
Posted Sep 19, 2025 - 21:17 UTC

Monitoring

We've identified a large increase in load which has now been resolved, and we're monitoring to ensure request latency returns to normal operating levels.
Posted Sep 19, 2025 - 21:07 UTC

Identified

We have discovered a rogue puma process contributing to the high latency and have restarted it. We are now investigating other processes to determine if they are exhibiting the same rogue behaviour.
Posted Sep 19, 2025 - 20:30 UTC

Investigating

We are investigating increased latency and timeouts with the REST and GraphQL endpoints.
Posted Sep 19, 2025 - 19:47 UTC
← Current Status Powered by Atlassian Statuspage