Job processing and scheduling not functioning

Incident Report for Buildkite

Resolved

The backlog has been fully processed and the system is back to normal. Apologies for the interruption.
Posted Jan 04, 2021 - 23:34 UTC

Monitoring

The issue has been identified, and a fix deployed. We are keeping an eye on the background job queues as they process the backlog.
Posted Jan 04, 2021 - 23:29 UTC

Investigating

We are currently investigating an issue affecting job processing and background jobs
Posted Jan 04, 2021 - 23:19 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications) and Job Queue, SCM Integrations.