Degraded performance for Notifications

Incident Report for Buildkite

Resolved

We have workaround in place which resolved the issue and work is in progress for permanent fix.

Posted Feb 14, 2023 - 00:30 UTC

Update

We have identified that the notifications workers are not scaling correctly. We've manually pinned them at max capacity while we continue to investigate

Posted Feb 14, 2023 - 00:22 UTC

Identified

Our team identified there is degraded performance for notifications and are continuing to investigate it

Posted Feb 14, 2023 - 00:12 UTC

This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Hipchat Notifications, Webhook Notifications).