Done! We're sorry it took so long. We had run multiple test upgrades throughout the week without a hitch, this one presented some new challenges that took a little extra time to work through.
Buildkite is back and everything should be running smoothly again! If you notice jobs stuck in weird states, a "cancel" + "retry" should get them going again. If that doesn't work, reach out to us hello@buildkite.com and we'll take a look.
Posted Apr 14, 2019 - 04:20 UTC
Update
We're finishing up now, and our graphs are starting to look again.
Posted Apr 14, 2019 - 03:51 UTC
Update
We've finished our maintenance tasks and services are coming online. You may see a few errors while our servers start warming up.
Posted Apr 14, 2019 - 03:25 UTC
Update
We've just marked our service as having "major outage" to indicate that nothing is working. We're still running our final tasks.
We'll post an update as soon as we're back!
Posted Apr 14, 2019 - 02:45 UTC
Identified
We're continuing our database upgrade. StatusPage automatically finished our previous scheduled maintenance incident before we had finished our maintenance tasks, so we'll post updates in this incident moving forward.
Posted Apr 14, 2019 - 02:05 UTC
This incident affected: Web, Agent API, REST API, Job Queue, Third Party Services (AWS ec2-us-east-1, AWS elasticache-us-east-1, AWS elb-us-east-1, AWS rds-us-east-1, PagerDuty Notification Delivery), SCM Providers (GitHub, Atlassian Bitbucket SSH, Atlassian Bitbucket Website and API, Atlassian Bitbucket Git via HTTPS), and Notifications (GitHub Commit Status Notifications, Email Notifications, Slack Notifications, Webhook Notifications).