High job dispatch latency
Incident Report for Buildkite
Postmortem

This was one of five incidents with a common root cause. The post mortem is available here.

Posted Jun 23, 2022 - 07:39 UTC

Resolved
This incident has been resolved, and the system is performing normally. Storage latency on a core database was elevated, and we worked closely with our vendor to resolve it.
Posted May 07, 2022 - 13:05 UTC
Update
The issue has been identified and service is beginning to return to normal.
Posted May 07, 2022 - 12:30 UTC
Update
We are continuing to investigate degraded queue performance.
Posted May 07, 2022 - 12:08 UTC
Investigating
We are seeing high dispatch latency, we are investigating.
Posted May 07, 2022 - 11:30 UTC
This incident affected: Job Queue.