Increased Web, API, and Agent API latencies

Incident Report for Buildkite

Resolved

This incident has been resolved.

Posted Apr 17, 2023 - 01:00 UTC

Monitoring

We've restarted all of our background workers and they're in the process of catching up, we are monitoring our service notifications service.

Posted Apr 17, 2023 - 00:54 UTC

Investigating

We're investigating increased latency on our background queues that will be leading to increased job dispatch times and other increased latencies on other Web, API and Agent API operations.

Posted Apr 17, 2023 - 00:49 UTC

This incident affected: Web, Agent API, REST API, Job Queue and Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Hipchat Notifications, Webhook Notifications).