We've rolled out fixes to our production systems which have resulted in backend queries running significantly faster. Our systems have all caught up on all scheduled workloads and everything is running smoothly again.
Sep 8, 09:52 AEST
Jobs are uploading and processing as normal now. Delays have almost reduced to zero.
Sep 8, 09:32 AEST
We’re seeing delays as the system catches up with the delayed workload, but we believe we have a handle on the problem.
Sep 8, 09:15 AEST
We’re seeing some pipelines uploading, and jobs slowly resuming processing. We’re still digging into the issue.
Sep 8, 08:41 AEST
The incident appears to be affecting the job queue in addition to the Agent API
Sep 8, 08:34 AEST
We are investigating pipelines failing to upload
Sep 8, 08:24 AEST