Delayed outbound notifications
Incident Report for Buildkite
This incident has been resolved.
Posted Oct 18, 2023 - 18:57 UTC
We have increased capacity and new notifications are being processed normally. Notifications that occurred during the incident are expected to be delivered but with a small delay.
Posted Oct 18, 2023 - 18:51 UTC
We are experiencing high load which is in excess of our rate internal rate limits. We are increasing capacity to accomodate the increased load. We will provide an update within 15 minutes.
Posted Oct 18, 2023 - 18:34 UTC
We're seen an increase in notification delivery latency of over 20 minutes. We are working to identify a cause. We will provide an update in 15 minutes.
Posted Oct 18, 2023 - 18:25 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications).