Delayed outbound notifications

Incident Report for Buildkite

Resolved

This incident has been resolved.

Posted Oct 18, 2023 - 18:57 UTC

Monitoring

We have increased capacity and new notifications are being processed normally. Notifications that occurred during the incident are expected to be delivered but with a small delay.

Posted Oct 18, 2023 - 18:51 UTC

Identified

We are experiencing high load which is in excess of our rate internal rate limits. We are increasing capacity to accomodate the increased load. We will provide an update within 15 minutes.

Posted Oct 18, 2023 - 18:34 UTC

Investigating

We're seen an increase in notification delivery latency of over 20 minutes. We are working to identify a cause. We will provide an update in 15 minutes.

Posted Oct 18, 2023 - 18:25 UTC

This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications).