Increased API error rate and latency
Incident Report for Buildkite
Resolved
After some consideration, we won't be reprocessing inbound webhooks received during 04:25-05:40, 11 Nov 2021 UTC.

Given that newer webhooks have been processed, we are worried that processing older webhooks may have negative consequences for deployment pipelines.

You can use the “New Build” button to trigger builds for missed commits if you need.

We’re so sorry for the interruption this has caused. Please reach out to support@buildkite.com if you need assistance.
Posted Nov 11, 2021 - 06:44 UTC
Update
Dispatches and other build events from during the queue unavailability have been reprocessed; we’re currently working on handling webhooks that we received during 04:25-05:40, 11 Nov 2021 UTC.

Outbound notifications from Buildkite to your commit statuses, Slack, custom webhook handlers etc from this time window will not be sent, but notifications for activity since then are being sent.

If you notice any stuck builds or have any further questions, please contact Buildkite Support at support@buildkite.com.
Posted Nov 11, 2021 - 06:25 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 11, 2021 - 05:43 UTC
Update
We are currently having an issue with the Redis job queue, which has become unavailable and the team is in the process of replacing it.
Posted Nov 11, 2021 - 05:20 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Nov 11, 2021 - 04:55 UTC
Update
We are continuing to investigate this issue.
Posted Nov 11, 2021 - 04:39 UTC
Investigating
We are currently investigating this issue.
Posted Nov 11, 2021 - 04:37 UTC
This incident affected: Third Party Services (AWS ec2-us-east-1, AWS elasticache-us-east-1, AWS elb-us-east-1, AWS rds-us-east-1, PagerDuty Notification Delivery), Web, Agent API, REST API, Job Queue, SCM Integrations, SCM Providers (GitHub, GitHub API Requests, Atlassian Bitbucket SSH, Atlassian Bitbucket Website and API, Atlassian Bitbucket Git via HTTPS), and Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Hipchat Notifications, Webhook Notifications).