Notification delays
Incident Report for Buildkite
Resolved
We have finished processing existing notifications from the backlog, and notifications are now working as expected.

With further investigation, we can see that GitHub Commit Statuses did not appear to be affected in this incident, but EventBridge was affected. We apologize for any confusion that this may have caused.
Posted Nov 17, 2021 - 01:52 UTC
Monitoring
We are using the additional capacity to process the existing notifications as we work through the backlog.
Posted Nov 17, 2021 - 01:33 UTC
Identified
The issue has been identified and a fix is being implemented.

We are also adding additional capacity to our notifications workers to work through existing notifications.
Posted Nov 17, 2021 - 01:24 UTC
Investigating
We are currently investigating this issue.
Posted Nov 17, 2021 - 00:58 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications).