Intermittent notification delays
Incident Report for Buildkite
Resolved
We've tracked down the problem to a set of notifications that were causing high internal load in our background processor. We've disabled them and we're working on some alternatives to ensure this doesn't happen in the future.
Posted 9 days ago. Jan 08, 2019 - 11:00 AEDT
Investigating
We've had some reports of intermittent delays with notifications (web hooks, GitHub commit statuses, etc). Our current systems aren't reporting any issues at the moment, but there seems to be a spike in delays every 12 hours or so. We're investigating the issue now.
Posted 9 days ago. Jan 08, 2019 - 10:11 AEDT
This incident affected: Notifications (Github Commit Status Notifications, Email Notifications, Slack Notifications, Hipchat Notifications, Pusher Pusher REST API, Pusher WebSocket client API, Webhook Notifications).