Continued Database Upgrade
Incident Report for Buildkite
Resolved
Done! We're sorry it took so long. We had run multiple test upgrades throughout the week without a hitch, this one presented some new challenges that took a little extra time to work through.

Buildkite is back and everything should be running smoothly again! If you notice jobs stuck in weird states, a "cancel" + "retry" should get them going again. If that doesn't work, reach out to us hello@buildkite.com and we'll take a look.
Posted 6 months ago. Apr 14, 2019 - 14:20 AEST
Update
We're finishing up now, and our graphs are starting to look again.
Posted 6 months ago. Apr 14, 2019 - 13:51 AEST
Update
We've finished our maintenance tasks and services are coming online. You may see a few errors while our servers start warming up.
Posted 6 months ago. Apr 14, 2019 - 13:25 AEST
Update
We've just marked our service as having "major outage" to indicate that nothing is working. We're still running our final tasks.

We'll post an update as soon as we're back!
Posted 6 months ago. Apr 14, 2019 - 12:45 AEST
Identified
We're continuing our database upgrade. StatusPage automatically finished our previous scheduled maintenance incident before we had finished our maintenance tasks, so we'll post updates in this incident moving forward.
Posted 6 months ago. Apr 14, 2019 - 12:05 AEST
This incident affected: SCM Providers (GitHub, Atlassian Bitbucket SSH, Atlassian Bitbucket Website and API, Atlassian Bitbucket Git via HTTPS), Web, Agent API, REST API, Job Queue, Third Party Services (AWS ec2-us-east-1, AWS elasticache-us-east-1, AWS elb-us-east-1, AWS rds-us-east-1, PagerDuty Notification Delivery), and Notifications (Github Commit Status Notifications, Email Notifications, Slack Notifications, Hipchat Notifications, Pusher Pusher REST API, Pusher WebSocket client API, Webhook Notifications).