Elevated error rates in pipeline uploads
Incident Report for Buildkite
Postmortem

On the 31st May at 05:27 UTC we deployed a change that removed a dependency we no longer needed. Unfortunately, we were using a transitive dependency of the library we removed to verify the safety of user provided regular expressions. This caused creating builds and uploading pipelines to fail if they contained conditionals with regular expressions. This transitive dependency was also included via other paths in our dev and test environments so it was only discovered in production. We rolled back the deployment and added an explicit dependency on the library. The incident was resolved at 06:05 (38 minutes).

Posted Jun 01, 2022 - 04:46 UTC

Resolved
This incident has been resolved.
Posted May 31, 2022 - 06:35 UTC
Monitoring
A deployment removed a dependency that was necessary but not properly declared. The on-call engineer identified the problem and reverted the change. Errors have reduced and we are continuing to monitor error rates.
Posted May 31, 2022 - 06:15 UTC
This incident affected: Agent API.