Elevated Latency on Notifications

Incident Report for Buildkite

Resolved

We continue to implement further mitigations to improve notification latency
Posted Feb 02, 2023 - 21:53 UTC

Update

Latency is now negligible for all notification types. We are continuing to actively monitor the situation.
Posted Feb 02, 2023 - 21:03 UTC

Update

We continue to see notification latency improve, and we are continuing to actively monitor the situation
Posted Feb 02, 2023 - 20:35 UTC

Monitoring

We're seeing notification latency improve, but we continue to proactively throttle some notification types
Posted Feb 02, 2023 - 20:00 UTC

Investigating

We are seeing elevated latency of up to an hour on step-level statuses. These have been deprioritized in order to manage latency across other notification types.

We continue to investigate the underlying root causes.
Posted Feb 02, 2023 - 19:30 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications).