Delayed notifications

This incident was the result of a lack of capacity in the group of workers that services outbound notifications queues. Notification queue latency reached 8 minutes due to this lack of capacity. During a blue/green rollout of these workers, autoscaling was manually disabled to increase the number of workers serving the queue in preparation for migrating the processing of background jobs between the blue and green side of the deployment. After shifting background workload across to the new workers, autoscaling was not re-enabled which prevented the deployment from scaling up to maximum capacity. We’re improving the documentation and tooling involved in this blue/green deployment process to prevent a recurrence of this issue and improve visibility into the status of autoscaling.