Resolved -
This incident has been resolved. All services are operating normally and remaining jobs have been processed.
We apologise for the disruption and thank you for your patience.
Nov 10, 12:00 UTC
Update -
All services are currently stable and we continue to monitor system status. We are working to unblock some remaining jobs that were impacted by the incident.
Thank you for your continued patience.
Nov 10, 10:46 UTC
Monitoring -
The issue has been resolved and service has been restored to all remaining customers. We are continuing to monitor to ensure stability.
We apologize for the disruption and thank you for your patience.
Nov 10, 09:02 UTC
Update -
We have restored service for additional customers and are continuing to work on a resolution for remaining affected customers.
We've identified the root cause of the issue: a database index on our annotations table was replaced, but the replacement index isn't being utilized as expected under load, which is causing performance degradation.
Thank you for your understanding and patience as we work to resolve this as quickly as possible.
Nov 10, 08:46 UTC
Update -
We have restored service for additional customers and are continuing to work on a resolution for remaining affected customers.
Nov 10, 08:05 UTC
Update -
We are actively working on a resolution for remaining customers. We do not have an estimated time for resolution at this time.
Nov 10, 07:24 UTC
Update -
We have restored service for the majority of customers and are continuing to work on those that remain.
We are working through applying a fix but the exact time to fully restored service is not yet clear.
Nov 10, 06:36 UTC
Identified -
The REST and GraphQL APIs, as well as the Web UI, are currently affected by increased error rates and slow performance.
We have identified the cause of the issue, and are working to address it.
Nov 10, 05:30 UTC
Investigating -
We have noticed issues with displaying build annotations, which may cause viewing builds in the web UI. We are still investigating and will provide another update shortly.
Nov 10, 05:22 UTC