Resolved -
This incident has now been resolved.
Feb 1, 03:30 UTC
Monitoring -
We’re confident in the status of all systems across Buildkite and are monitoring the situation in order to ensure it remains healthy.
Feb 1, 02:40 UTC
Update -
Build notifications and commit statuses continue to be delayed by approximately 20 minutes, however we expect to see this resolved within the next 2 hours.
Feb 1, 02:05 UTC
Update -
Latency for inbound webhook processing and trigger jobs remains acceptable levels. We continue working to improve the delays to post build notifications, including commit statuses.
Feb 1, 01:34 UTC
Update -
Latency for inbound webhook processing and trigger jobs has returned to acceptable levels. We are still working to improve the delays to post build notifications, including commit statuses.
Feb 1, 01:07 UTC
Update -
The mitigations have taken effect and latency for webhook processing and trigger jobs has returned to normal levels. We are still experiencing delays to post build notifications, including commit statuses and are working to improve that now.
Feb 1, 00:32 UTC
Update -
The mitigations we’ve rolled out have further decreased asynchronous queue latency, we are now re-allocating a number of asynchronous jobs to lower priority queues to attempt to further improve the latency of the queue processing trigger jobs, posting of commit statuses, and ingesting webhooks.
Jan 31, 23:55 UTC
Update -
We're continuing to implement mitigations to process trigger jobs, posting of commit statuses, and ingesting webhooks
Jan 31, 23:23 UTC
Identified -
Our mitigations are taking effect and delays have reduced to about 5 minutes on updating trigger jobs, posting commit statuses, and ingesting webhooks
Jan 31, 22:53 UTC
Update -
We are continuing to investigate the issue. We are observing delays of about 8 minutes on updating trigger jobs, posting commit statuses, and ingesting webhooks
Jan 31, 22:04 UTC
Update -
We are pausing non-time-sensitive work to allocate additional resources to critical workloads
Jan 31, 21:25 UTC
Update -
Based on our current understanding of the incident we have commenced a restart of our asynchronous background workers to attempt to reduce queue processing times.
Jan 31, 21:02 UTC
Update -
We are continuing to investigate this issue.
Jan 31, 20:40 UTC
Update -
We are continuing to investigate this issue.
Jan 31, 20:22 UTC
Update -
We're investigating reports of elevated processing times for some services and features
Jan 31, 20:08 UTC
Update -
We are continuing to investigate this issue.
Jan 31, 20:05 UTC
Investigating -
We're investigating reports of elevated processing times for some services and features
Jan 31, 20:04 UTC