Elevated processing times

Incident Report for Buildkite

Resolved

This incident has now been resolved.
Posted Feb 01, 2023 - 03:30 UTC

Monitoring

We’re confident in the status of all systems across Buildkite and are monitoring the situation in order to ensure it remains healthy.
Posted Feb 01, 2023 - 02:40 UTC

Update

Build notifications and commit statuses continue to be delayed by approximately 20 minutes, however we expect to see this resolved within the next 2 hours.
Posted Feb 01, 2023 - 02:05 UTC

Update

Latency for inbound webhook processing and trigger jobs remains acceptable levels. We continue working to improve the delays to post build notifications, including commit statuses.
Posted Feb 01, 2023 - 01:34 UTC

Update

Latency for inbound webhook processing and trigger jobs has returned to acceptable levels. We are still working to improve the delays to post build notifications, including commit statuses.
Posted Feb 01, 2023 - 01:07 UTC

Update

The mitigations have taken effect and latency for webhook processing and trigger jobs has returned to normal levels. We are still experiencing delays to post build notifications, including commit statuses and are working to improve that now.
Posted Feb 01, 2023 - 00:32 UTC

Update

The mitigations we’ve rolled out have further decreased asynchronous queue latency, we are now re-allocating a number of asynchronous jobs to lower priority queues to attempt to further improve the latency of the queue processing trigger jobs, posting of commit statuses, and ingesting webhooks.
Posted Jan 31, 2023 - 23:55 UTC

Update

We're continuing to implement mitigations to process trigger jobs, posting of commit statuses, and ingesting webhooks
Posted Jan 31, 2023 - 23:23 UTC

Identified

Our mitigations are taking effect and delays have reduced to about 5 minutes on updating trigger jobs, posting commit statuses, and ingesting webhooks
Posted Jan 31, 2023 - 22:53 UTC

Update

We are continuing to investigate the issue. We are observing delays of about 8 minutes on updating trigger jobs, posting commit statuses, and ingesting webhooks
Posted Jan 31, 2023 - 22:04 UTC

Update

We are pausing non-time-sensitive work to allocate additional resources to critical workloads
Posted Jan 31, 2023 - 21:25 UTC

Update

Based on our current understanding of the incident we have commenced a restart of our asynchronous background workers to attempt to reduce queue processing times.
Posted Jan 31, 2023 - 21:02 UTC

Update

We are continuing to investigate this issue.
Posted Jan 31, 2023 - 20:40 UTC

Update

We are continuing to investigate this issue.
Posted Jan 31, 2023 - 20:22 UTC

Update

We're investigating reports of elevated processing times for some services and features
Posted Jan 31, 2023 - 20:08 UTC

Update

We are continuing to investigate this issue.
Posted Jan 31, 2023 - 20:05 UTC

Investigating

We're investigating reports of elevated processing times for some services and features
Posted Jan 31, 2023 - 20:04 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications) and Job Queue, SCM Integrations.