Intermittent notification delays

Incident Report for Buildkite

Resolved

We've tracked down the problem to a set of notifications that were causing high internal load in our background processor. We've disabled them and we're working on some alternatives to ensure this doesn't happen in the future.

Posted Jan 08, 2019 - 00:00 UTC

Investigating

We've had some reports of intermittent delays with notifications (web hooks, GitHub commit statuses, etc). Our current systems aren't reporting any issues at the moment, but there seems to be a spike in delays every 12 hours or so. We're investigating the issue now.

Posted Jan 07, 2019 - 23:11 UTC

This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications).