We believe this issue should now be resolved! 🎉 We'll be keeping an eye on things. 👀
Yesterday we noticed some increased agent traffic and increased our autoscaling capacity to deal with the load, and it seemed to work as expected.
Today jobs started slowing down. Our main database and web servers were fine, but our job log chunk database couldn't quite keep up with the load. This was having some follow on effects becoming degraded performance on the agent endpoint and causing build scheduling and running delays. We've added another shard to deal with this increased load, and performance seems to be back to normal.
Posted Oct 18, 2016 - 18:38 UTC
Update
We've identified the issue, restored immediate performance, and are rolling out more infrastructure to handle this increased load in the future.
Posted Oct 18, 2016 - 16:36 UTC
Identified
High agent load is causing some delays while running builds. We're looking into it.