We’re confident in the status of all systems across Buildkite and are monitoring the situation in order to ensure it remains healthy.
Posted Feb 01, 2023 - 02:40 UTC
Update
Build notifications and commit statuses continue to be delayed by approximately 20 minutes, however we expect to see this resolved within the next 2 hours.
Posted Feb 01, 2023 - 02:05 UTC
Update
Latency for inbound webhook processing and trigger jobs remains acceptable levels. We continue working to improve the delays to post build notifications, including commit statuses.
Posted Feb 01, 2023 - 01:34 UTC
Update
Latency for inbound webhook processing and trigger jobs has returned to acceptable levels. We are still working to improve the delays to post build notifications, including commit statuses.
Posted Feb 01, 2023 - 01:07 UTC
Update
The mitigations have taken effect and latency for webhook processing and trigger jobs has returned to normal levels. We are still experiencing delays to post build notifications, including commit statuses and are working to improve that now.
Posted Feb 01, 2023 - 00:32 UTC
Update
The mitigations we’ve rolled out have further decreased asynchronous queue latency, we are now re-allocating a number of asynchronous jobs to lower priority queues to attempt to further improve the latency of the queue processing trigger jobs, posting of commit statuses, and ingesting webhooks.
Posted Jan 31, 2023 - 23:55 UTC
Update
We're continuing to implement mitigations to process trigger jobs, posting of commit statuses, and ingesting webhooks
Posted Jan 31, 2023 - 23:23 UTC
Identified
Our mitigations are taking effect and delays have reduced to about 5 minutes on updating trigger jobs, posting commit statuses, and ingesting webhooks
Posted Jan 31, 2023 - 22:53 UTC
Update
We are continuing to investigate the issue. We are observing delays of about 8 minutes on updating trigger jobs, posting commit statuses, and ingesting webhooks
Posted Jan 31, 2023 - 22:04 UTC
Update
We are pausing non-time-sensitive work to allocate additional resources to critical workloads
Posted Jan 31, 2023 - 21:25 UTC
Update
Based on our current understanding of the incident we have commenced a restart of our asynchronous background workers to attempt to reduce queue processing times.
Posted Jan 31, 2023 - 21:02 UTC
Update
We are continuing to investigate this issue.
Posted Jan 31, 2023 - 20:40 UTC
Update
We are continuing to investigate this issue.
Posted Jan 31, 2023 - 20:22 UTC
Update
We're investigating reports of elevated processing times for some services and features
Posted Jan 31, 2023 - 20:08 UTC
Update
We are continuing to investigate this issue.
Posted Jan 31, 2023 - 20:05 UTC
Investigating
We're investigating reports of elevated processing times for some services and features
Posted Jan 31, 2023 - 20:04 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Hipchat Notifications, Webhook Notifications) and Job Queue, SCM Integrations.