We observed degraded performance on the Agent API from 16-11-21 20:20 UTC to 16-11-21 21:35 UTC due to high database load. This load was caused by contention between normal operations (Job Dispatch, API, Dashboard UI) and a background notification worker that concurrently accesses the jobs and job event tables to generate Github commit status notifications. In order to reduce the high database load, we temporarily reduced the capacity of the background notification worker queue, thereby limiting the number of workers that were concurrently accessing the contended tables. This reduced the observed latency on the Agent API. We are implementing application level concurrency limiting on the background worker to prevent this contention recurring at the database layer. Unfortunately, temporarily reducing the capacity of our notification worker queue later resulted in increased latency for service notifications, see https://www.buildkitestatus.com/incidents/q4104hnbyvsx.