We believe the performance issues have abated and that things are operating as normal now.
Posted Sep 25, 2020 - 00:25 UTC
Monitoring
We believe the database issue causing slow Agent API responses and Job Queue processing has been addressed, and response and dispatch times are now improving.
Posted Sep 24, 2020 - 22:42 UTC
Update
GitHub commit statuses are being published normally again. Apologies for the interruption.
Posted Sep 24, 2020 - 22:38 UTC
Update
GitHub commit statuses are currently failing to be sent. We are deploying a fix. No statuses have been lost, and they will be dispatched once the fix is deployed.
Posted Sep 24, 2020 - 22:32 UTC
Investigating
We’re continuing to investigate the root causes of the Agent API and Job Queue slowness, which has not yet abated.
Posted Sep 24, 2020 - 22:21 UTC
Identified
We believe we have identified the cause of the slow responses and that they have begun to speed back up as the system works to catch up.
Posted Sep 24, 2020 - 21:57 UTC
Investigating
We’re seeing some slowness in both the Buildkite web UI and the Agent API. Some jobs are slow to be assigned or accepted, and we’re currently investigating the cause.
Posted Sep 24, 2020 - 21:04 UTC
This incident affected: Notifications (GitHub Commit Status Notifications) and Agent API, Job Queue.