Elevated Latency on Notifications
Incident Report for Buildkite
Resolved
We continue to implement further mitigations to improve notification latency
Posted Feb 02, 2023 - 21:53 UTC
Update
Latency is now negligible for all notification types. We are continuing to actively monitor the situation.
Posted Feb 02, 2023 - 21:03 UTC
Update
We continue to see notification latency improve, and we are continuing to actively monitor the situation
Posted Feb 02, 2023 - 20:35 UTC
Monitoring
We're seeing notification latency improve, but we continue to proactively throttle some notification types
Posted Feb 02, 2023 - 20:00 UTC
Investigating
We are seeing elevated latency of up to an hour on step-level statuses. These have been deprioritized in order to manage latency across other notification types.

We continue to investigate the underlying root causes.
Posted Feb 02, 2023 - 19:30 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications).