High Error Rates Across All Components
Incident Report for Buildkite
Postmortem

This was one of five incidents with a common root cause. The post mortem is available here.

Posted Jun 23, 2022 - 07:39 UTC

Resolved
We have been advised by the vendor that this issue has been fixed and performance has returned to normal. This incident is now resolved.
Posted May 10, 2022 - 22:32 UTC
Monitoring
We are seeing performance return to normal and seeing builds running again. We are confirming the root cause has been fixed.
Posted May 10, 2022 - 22:25 UTC
Identified
We've identified an issue with our primary database affecting write latency. We are working with the vendor to resolve the issue.
Posted May 10, 2022 - 22:17 UTC
Update
We are continuing to investigate this issue.
Posted May 10, 2022 - 22:04 UTC
Investigating
We are currently investigating this issue.
Posted May 10, 2022 - 22:04 UTC
This incident affected: Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Hipchat Notifications, Webhook Notifications) and Web, Agent API, REST API, Job Queue, SCM Integrations.