High job upload and process latency
Incident Report for Buildkite
On May 23 at 04:16 UTC a feature was turned on that caused unexpected errors in uploading job logs for approximately 3300 builds.
The effect was short delays in processing job uploads. Our engineers were automatically paged at 04:22 UTC when job logs upload latency exceeded 2 seconds.
The feature was then immediately disabled and job processing began to improve. It returned to within normal limits at 04:25 UTC and all builds caught up.
We are continuing to monitor. Please contact us if you have any issues.
Posted May 23, 2022 - 04:16 UTC