Increased error rates on some web requests

Incident Report for Buildkite

Resolved

Network performance and requests are performing normally again, and our infrastructure upgrades are in place to prevent this from happening again. We’re sorry for the inconvenience and appreciate your patience 🙏
Posted Aug 29, 2017 - 23:23 UTC

Monitoring

Network upgrades are complete, everything is available and responding quickly again. Job logs response times are back to normal. We’re continuing to monitor the performance of the network upgrades.
Posted Aug 29, 2017 - 23:07 UTC

Update

We’re performing further network upgrades, which may cause job logs and websocket notifications to be temporarily unavailable. We’ll keep you updated on the progress.
Posted Aug 29, 2017 - 22:47 UTC

Update

We’re performing unplanned network maintenance in order to improve S3 and web socket connectivity issues that have been causing delays in the web interface. Job logs and push notifications will be unavailable for approximately 5 minutes.
Posted Aug 29, 2017 - 22:28 UTC

Identified

Some of the instances serving the web interface were returning errors, and have been replaced. We’re continuing to monitor performance and error rates, and will keep you updated.
Posted Aug 29, 2017 - 22:19 UTC

Investigating

We are currently investigating this issue.
Posted Aug 29, 2017 - 21:52 UTC