Agent API Connection Issues

Incident Report for Buildkite

Resolved

Graphs have returned to normal, and jobs and agents should be running again as normal. Please reach out to support if you have any stuck jobs

Posted Oct 16, 2019 - 01:00 UTC

Monitoring

We've removed the database that was having the issue, and graphs are starting to look better now. We're still all hands on deck, and keeping an eye on systems.

Posted Oct 16, 2019 - 00:22 UTC

Update

We've identified an issue with one of the databases responsible for storing log chunks from agents. We've started the process to decommission that database, which we hope will bring things back to normal.

Posted Oct 15, 2019 - 23:59 UTC

Update

We are continuing to investigate this issue.

Posted Oct 15, 2019 - 23:42 UTC

Investigating

We've received reports that some agents are loosing connections and builds are hanging. We're looking into what the issue might be.

Posted Oct 15, 2019 - 23:32 UTC

This incident affected: Agent API and Job Queue.