Pipelines failing to process and upload
Incident Report for Buildkite
Resolved
We've rolled out fixes to our production systems which have resulted in backend queries running significantly faster. Our systems have all caught up on all scheduled workloads and everything is running smoothly again.
Posted Sep 07, 2018 - 23:52 UTC
Monitoring
Jobs are uploading and processing as normal now. Delays have almost reduced to zero.
Posted Sep 07, 2018 - 23:32 UTC
Identified
We’re seeing delays as the system catches up with the delayed workload, but we believe we have a handle on the problem.
Posted Sep 07, 2018 - 23:15 UTC
Update
We’re seeing some pipelines uploading, and jobs slowly resuming processing. We’re still digging into the issue.
Posted Sep 07, 2018 - 22:41 UTC
Update
The incident appears to be affecting the job queue in addition to the Agent API
Posted Sep 07, 2018 - 22:34 UTC
Investigating
We are investigating pipelines failing to upload
Posted Sep 07, 2018 - 22:24 UTC
This incident affected: Agent API and Job Queue.