Pipelines failing to process and upload

Incident Report for Buildkite

Resolved

We've rolled out fixes to our production systems which have resulted in backend queries running significantly faster. Our systems have all caught up on all scheduled workloads and everything is running smoothly again.

Posted Sep 07, 2018 - 23:52 UTC

Monitoring

Jobs are uploading and processing as normal now. Delays have almost reduced to zero.

Posted Sep 07, 2018 - 23:32 UTC

Identified

We’re seeing delays as the system catches up with the delayed workload, but we believe we have a handle on the problem.

Posted Sep 07, 2018 - 23:15 UTC

Update

We’re seeing some pipelines uploading, and jobs slowly resuming processing. We’re still digging into the issue.

Posted Sep 07, 2018 - 22:41 UTC

Update

The incident appears to be affecting the job queue in addition to the Agent API

Posted Sep 07, 2018 - 22:34 UTC

Investigating

We are investigating pipelines failing to upload

Posted Sep 07, 2018 - 22:24 UTC

This incident affected: Agent API and Job Queue.