Degraded performance and increased error rates

Incident Report for Buildkite

Resolved

We experienced an issue that caused a brief increase in errors for the Agent API. This also impacted latency for notifications. All notifications were stored in a queue and processed. Latency is now back to normal.
Posted Apr 08, 2026 - 23:12 UTC

Monitoring

We have identified and fixed the issue. We are monitoring and seeing signs of improvement.
Posted Apr 08, 2026 - 22:40 UTC

Investigating

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.
Posted Apr 08, 2026 - 22:26 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications) and Agent API.