Delayed outbound job event notifications

Incident Report for Buildkite

Resolved

This incident has been resolved.

Posted Oct 18, 2023 - 21:30 UTC

Monitoring

We have worked through the delayed job notifications. We will continue monitoring to ensure they are all processed successfully.

Posted Oct 18, 2023 - 21:26 UTC

Identified

We are continuing to work through the backlog of notifications.

Posted Oct 18, 2023 - 20:59 UTC

Investigating

We've identified a backlog of delayed job event notifications and are working through that backlog at the moment. All other notification events (including new job events) are unaffected.

Posted Oct 18, 2023 - 20:29 UTC

This incident affected: Notifications (GitHub Commit Status Notifications, Slack Notifications, Webhook Notifications).