Elevated latency

Incident Report for Buildkite

Resolved

This incident has been resolved.
Posted Apr 25, 2023 - 21:15 UTC

Monitoring

We have confirmed that latency has returned to normal, and we are monitoring the affected services as we investigate the root cause.
Posted Apr 25, 2023 - 21:01 UTC

Investigating

We are still seeing a high latency processing incoming webhooks and dispatching jobs to agents, and we are continuing to investigate the root cause
Posted Apr 25, 2023 - 20:29 UTC

Identified

The issue has been identified and we are currently working on reducing the load.
Posted Apr 25, 2023 - 20:04 UTC

Update

We are continuing to investigate this issue.
Posted Apr 25, 2023 - 19:41 UTC

Investigating

We are currently investigating an issue of elevated latency on some background queue processing, including processing incoming webhooks and dispatching jobs to agents.
Posted Apr 25, 2023 - 19:21 UTC
This incident affected: Job Queue.