Increased job dispatch latency
Incident Report for Buildkite
Postmortem

This incident was caused by similar circumstances to our January 26th (UTC) incident. Details of the cause and our ongoing mitigations can be found in the postmortem available at https://www.buildkitestatus.com/incidents/xb7h2m17fssf

Posted Jan 30, 2024 - 05:31 UTC
Resolved
We’ve applied the known mitigations and have seen an improvement in database query plan performance. We continue to work on a long term fix.
Posted Jan 30, 2024 - 00:46 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jan 30, 2024 - 00:30 UTC
Identified
We're investigating an issue which is causing increased job dispatch latency.
We suspect this is a re-occurrence of a previous incident, which is caused by poor Postgres query performance due to an inefficient query plan.
We're applying mitigations now to attempt to improve system performance
Posted Jan 30, 2024 - 00:18 UTC
This incident affected: Job Queue.
← Current Status Powered by Atlassian Statuspage