Pipelines failing to process and upload
Incident Report for Buildkite
Resolved
We've rolled out fixes to our production systems which have resulted in backend queries running significantly faster. Our systems have all caught up on all scheduled workloads and everything is running smoothly again.
Posted 2 months ago. Sep 08, 2018 - 09:52 AEST
Monitoring
Jobs are uploading and processing as normal now. Delays have almost reduced to zero.
Posted 2 months ago. Sep 08, 2018 - 09:32 AEST
Identified
We’re seeing delays as the system catches up with the delayed workload, but we believe we have a handle on the problem.
Posted 2 months ago. Sep 08, 2018 - 09:15 AEST
Update
We’re seeing some pipelines uploading, and jobs slowly resuming processing. We’re still digging into the issue.
Posted 2 months ago. Sep 08, 2018 - 08:41 AEST
Update
The incident appears to be affecting the job queue in addition to the Agent API
Posted 2 months ago. Sep 08, 2018 - 08:34 AEST
Investigating
We are investigating pipelines failing to upload
Posted 2 months ago. Sep 08, 2018 - 08:24 AEST
This incident affected: Agent API and Job Queue.