All times UTC unless stated otherwise.
Customers experienced elevated errors and latency across the Buildkite web interface, REST API, Agent API, and job queue during two periods: 18:43:00–19:21:00 and 20:02:00–20:28:00 on September 9, 2026.
During these periods, customers encountered failed API requests, delayed job dispatch, and errors in agent operations including authentication, job acceptance, and artifact uploads. Some jobs impacted by the incident did not recover automatically. These jobs had to be manually cancelled and retried to complete successfully.
Between 18:43:00–19:21:00: Agent API had an error rate up to 11.2% with no latency impact. REST API had a 9.7% error rate with no latency impact. Web request had a 2.6% error rate with no latency impact. Job queues were delayed by up to 3 minutes.
Between 20:02:00–20:28:00: Agent API had an error rate up to 24.7% with no latency impact. REST API had a 4.9% error rate with no latency impact. Web requests had a 0.8% error rate with no latency impact. Job queues were delayed by up to 10 minutes. During this window, notifications were also delayed up to 1 minute.
Core services fully recovered by 20:28:00.
Several Buildkite services use a shared Redis cluster for coordination, caching, and agent-facing operations. This is one of the last pieces of non-shard aligned shared infrastructure, and is on our roadmap to address. AWS enforces a connection limit of 65,000 connections per node on this Redis cluster.
As part of our ongoing migration from Amazon ECS to Amazon EKS, we reduced the number of Ruby threads in each Puma and Sidekiq process to their framework defaults. In ECS, we historically configured high thread counts, which caused resource contention and increased latency without a corresponding increase in throughput.
We’d intentionally timed this work as part of the EKS migration because the migration already required us to redesign how these services are sized and scaled: EKS can respond directly to request queues and worker utilization, allowing us to run more, smaller application containers while preserving total capacity. This gives customers faster, more predictable API responses and background-job processing, particularly during periods of high load.
We began testing Puma Agent services on EKS in July and migrated traffic incrementally from late July, completing the migration earlier this week. Running more, smaller processes increased total Redis connection demand—a trade-off we had modeled, but not adequately accounted for before completing the migration.
Before the incident, our shared services were already using much of the available connection capacity on the impacted Redis cluster, around 60-80% throughout a 24 hour window.
At 18:43:10 we started seeing elevated HTTP request errors. At this time, our application reported a burst of Redis connection timeouts, all targeting a single replica in the cluster. Clients with active Redis connections then attempted to reconnect. Failed connections were retried, causing a surge in connection attempts that exhausted that same Redis host’s connection-tracking allowance. This caused the host to reject new connections from the Agent API’s application pods, which led to increased request latency for our Puma Agent service.
At 18:43:30, the increased request latency from this network failure caused our application platform to rapidly start more Puma Agent containers to respond to the increase in request latency. Within a minute, we’d doubled our running pod count. Each additional container opened its own pool of Redis connections.
At 18:44:00, several Redis nodes hit their limit of 65,000 connections, and latency of requests slowed further triggering more Puma Agent autoscaling.
Once the Redis connection limits were reached, applications could no longer reliably connect to Redis. This caused failures across agent authentication, job assignment, artifact uploads, and other API operations.
The failures then created a feedback loop: slower requests caused our shard aligned Puma Agent services to autoscale, and those additional containers opened more Redis connections, placing further pressure on Redis.
By 18:50:00, our system stabilized at max, running about five times the number of pods we’d started with. This stable state allowed the autoscaling to start bringing pod counts back down.
By 19:10:00 the error rate had returned to 0 and and services were no longer degraded.
By 19:20:00 our pod counts had autoscaled down to more reasonable but still high numbers, and the Redis cluster was functioning again. At this point customers saw recovery.
At 20:02:00, a routine application deployment temporarily introduced additional containers. Since the Redis cluster was already under strain, this quickly increased request latency which added autoscaling on top of the regular deployment surge numbers. This amplified the connection growth again and caused the second period of degraded service, lasting until 20:28:00.
The influx of new pods resulted in a very large number of Envoy Gateway configuration updates, which in turn caused some network throttling of the underlying node. This caused packet drops and delays updating our Cilium Operator, which controls configuration for the cluster’s overlay network. This caused resulted some Buildkite Jobs to remain in a stuck state for some customers, requiring manual retries.
When we saw that the Redis cluster had reached its capacity to handle incoming connections, we immediately began adding more nodes to the cluster, increasing incoming connection capacity by 33%.
When we saw the impact of deployments, we immediately paused further application deployments while we investigated the connection growth.
We increased the Puma Agent utilization autoscaling threshold, reducing the rate at which workloads scale in response to brief load increases. We also increased our scale up stabilization window to prevent future runaway autoscaling of our workloads.
We reduced the configured maximum surge of our workloads during application deployments, to reduce pod churn impact on connections.
Since Redis connections and latency remained stable, we then resumed deployments while continuing to monitor the cluster.
Reduce autoscaling sensitivity.
Reduce deployment-related load.
Increase immediate Redis capacity by 33%.
Reduce connection pressure from high-traffic workloads.
Bound aggregate Redis connection demand.