Overview

On Friday 8th December between 5:00PM UTC and 7:00PM UTC some builds were not processing correctly and did not finish as expected. This manifested to customers as builds not being marked as completed when they should be. Our monitoring failed to pick this up and we were alerted by a customer at 6:45PM UTC. We traced this to a combination of flaws in our background job monitoring system and job retry system. At 7:00PM UTC the issue was resolved once we removed the host and corrected all affected builds.

What was Affected

• Some builds and jobs did not finish or progress correctly — they appeared "stuck" in the web dashboard
• Some notifications may not have been delivered
• Some webhooks may not have been processed

Root Cause

A new background job machine configuration was put into production on a test instance, but had a misconfigured pgbouncer configuration that didn't allow for enough database connections. In addition, this type of worker failure bypassed our standard exception catching and retry behaviour, causing background jobs to be lost without retry or error reporting.

Timeline

6th December

10PM UTC — As part of a plan to increase capacity in our background job workers, we investigated running more background job workers on each host in our cluster. Capacity seemed sufficient to go from one to two processes per host. We tested this during Australian peak load for short periods and all metrics indicated stability.

7th December

10AM UTC — We rolled out a single new host running an extra worker process as a canary. This was alongside our regular worker hosts which were still operating as normal.

2:54PM UTC — The new host started intermittently failing background jobs due to upstream database connection pooling issues.

5:00PM UTC — The new host hit peak capacity and almost all background jobs processed by this host failed. Under load, this particular error manifested in a way that prohibited automatic retry and started dropping background jobs, so did not self-heal. This was due to the errors occurring in the middleware portion of the queue system, which was incorrectly handling errors.

6:45PM UTC — We were woken up by a customer emergency page and started investigating.

6:54PM UTC — The new host's workers were stopped and background jobs success rates returned to normal.

7:00PM UTC — We manually processed all builds in the last few hours to fix any "stuck" builds, then spent some time verifying everything else in the system was in a sane state.

Changes We've Made

• More alerts have been configured for background job failures, including specific ones that would have alerted us as soon as the queue error rate on this host exceeded the norm.
• Our background job workers now use more resilient mechanisms which will further protect against dropping jobs
• Increased the capacity of our database pooler
• We've provided selected teams with emergency paging capabilities, as part of a larger future trial

Ongoing Improvements

• The long time to detect on this incident was caused by the lack of monitoring on this particular type of job failure, which meant that we didn't receive automated alerts and had to rely on an emergency page from a customer. This was of particular concern to us and we'll be reviewing our monitoring and alerting around job queues extensively.
• Developing a way for our Slack community to wake us up when something's going wrong
• Increasing the capacity of our primary database this weekend