Overview

On Thursday 30th November between 6:00PM UTC and 7:30PM UTC job log uploads were failing for a large majority of accounts, causing many build agents to block whilst waiting for their current job’s log to successfully upload. Once the job logs service recovered, agents successfully posted their logs and builds continued as normal.

What was Affected

• Job console output for jobs across most accounts were not visible. The web interface showed the job was "running" job but with no console output.
• Jobs and builds across most accounts were unusable (effectively stalled). The jobs had actually executed their commands, but then the agent just (indefinitely) stalled as it waited for all job log chunk uploads to finished before completing the job. This meant builds were also blocked if they were waiting for these jobs to complete.

Root Cause

In an effort to increase the responsiveness of job logs, we had prepared a new job log database to be put in production and had successfully tested it with a subset of Buildkite accounts. Even though we'd done this same operation many times in the past, we failed to recognise three important differences:

1. The new database was an AWS Aurora Postgres cluster, as opposed to the AWS RDS Postgres Multi-AZ instance, which needs to be operated in a very different manner in failure scenarios. Instead of treating a single multi-az instance as a cluster, you deal with a first-class "Cluster" concept, which has a single writer and as many readers as you wish.
2. The memory profile for AWS Aurora Postgres is wildly different for AWS RDS Postgres. Even though the new and old database had the same amount of memory, Aurora requires at least double the amount to handle the same number of connections. We've since discovered this is a known issue with Aurora Postgres.
3. The hostname for an Aurora cluster works slightly differently to an AWS Postgres multi-AZ instance. The difference caused us to incorrectly use the hostname of the primary "writer" instance instead of the cluster hostname. Had we used the cluster hostname, which always resolves to the current "writer", pgbouncer bouncer would have transparently reconnected to the new instance after the failover.

Graphs of the memory behaviour of an RDS Postgres db.m4.xlarge instance (Kratos) vs an Aurora Postgres db.r4.xlarge instance (Bia):

Aurora has a dramatically lower baseline of memory available, but seems to use less memory per additional connection.

Timeline

30th November

10:30AM UTC - We deployed code to allocate most new jobs to the new job log shard database. We monitored the database and overall systems, and everything looked okay.

5:41PM UTC - The new job log database performed a failover due to memory exhaustion, causing the secondary instance to become new primary ("writer") and the original host to become the secondary ("reader"). Even though the failover completed in 25 seconds, job logs were still failing to render and upload.

5:57PM UTC - Our team received a PagerDuty notification via BugSnag that there was an error spike writing to the job logs database.

6:33PM UTC - We upgraded the instance size of what we thought was the primary database, in the same way we had done previously with AWS RDS Postgres. But errors continued.

7:01PM UTC - The secondary on-call was alerted to provide assistance with debugging, allowing primary on-call to handle communication.

7:18PM UTC - We discovered the host name our pgbouncer instances were connecting to was not in fact the cluster hostname, but an instance hostname, causing all requests to go to the reader.

7:30PM UTC - All pgbouncer instances had been reconfigured to point to the correct hostname, and job log service was restored.

Changes We’ve Made

• Team education on the operational differences between AWS RDS Postgres and AWS Aurora Postgres, so we can more effectively manage Aurora databases under failure scenarios.
• Extra alerting and monitoring around available memory on the job log chunk databases.

Ongoing Improvements

• Slower introduction of new database systems into production. Horizontal shard scaling (which we support for job log chunks already) will be preferred for addressing urgent scaling requirements.
• Controlled testing of failover scenarios should be done with any new managed systems (in this case, testing and understanding AWS Aurora failovers, instance modifications, etc)
• Investigate a more peer-review-friendly process for modifying our database credentials configuration.
• Secondary on-call should have been notified earlier, to provide a fresh pair of eyes and/or to help with communications.
• Review the agent and web interface behaviour around job log chunk upload failure scenarios. For example, we should show an error message or indicate that log streaming from the agent has stalled.
• Work with AWS to debug the unusually low baseline of memory available on Aurora Postgres instances.

Please contact us if you have any questions or comments at hello@buildkite.com