Partial database outage

Incident Report for Buildkite

Resolved

All impacted systems are fully functional and have remained stable with the fix deployed. We will continue to monitor the situation, and will take steps to prevent it occurring again.

Posted Feb 26, 2019 - 07:33 UTC

Monitoring

We detected high error rates on the dashboard and agent APIs due to a partial database outage. We've rolled out a fix and all systems have returned to normal. We'll continue to all systems to ensure performance is fully restored.

Posted Feb 26, 2019 - 06:53 UTC

This incident affected: Web, Agent API, REST API, and Job Queue.