Notification delays

Incident Report for Buildkite

Resolved

We have finished processing existing notifications from the backlog, and notifications are now working as expected.

With further investigation, we can see that GitHub Commit Statuses did not appear to be affected in this incident, but EventBridge was affected. We apologize for any confusion that this may have caused.
Posted Nov 17, 2021 - 01:52 UTC

Monitoring

We are using the additional capacity to process the existing notifications as we work through the backlog.
Posted Nov 17, 2021 - 01:33 UTC

Identified

The issue has been identified and a fix is being implemented.

We are also adding additional capacity to our notifications workers to work through existing notifications.
Posted Nov 17, 2021 - 01:24 UTC

Investigating

We are currently investigating this issue.
Posted Nov 17, 2021 - 00:58 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications).