High job upload and process latency

Incident Report for Buildkite

Resolved

On May 23 at 04:16 UTC a feature was turned on that caused unexpected errors in uploading job logs for approximately 3300 builds.
The effect was short delays in processing job uploads. Our engineers were automatically paged at 04:22 UTC when job logs upload latency exceeded 2 seconds.
The feature was then immediately disabled and job processing began to improve. It returned to within normal limits at 04:25 UTC and all builds caught up.
We are continuing to monitor. Please contact us if you have any issues.

Posted 3 years ago. May 23, 2022 - 04:16 UTC