Elevated error rate on build creation
Incident Report for Buildkite
Resolved
A change to our database permissions caused creation of new builds to fail for some customers.
Affected customers trying to create new builds via the API would have received a 500 error.
Builds created via webhook (e.g. from Github), trigger steps or scheduled builds were delayed by up to 8 minutes.
Posted Oct 10, 2024 - 03:05 UTC
Monitoring
We have deployed a mitigation to fix the issue, and are now monitoring.

During the period of higher error rates, builds created via API may have failed outright. Builds created via webhooks, triggers or schedules are retried, though there may be some latency in processing these now due to retry back-off.
Posted Oct 10, 2024 - 02:41 UTC
Investigating
Our monitoring has detected an elevated error rate in creating builds. We're currently investigating the issue, and will provide an update soon.
Posted Oct 10, 2024 - 02:30 UTC
This incident affected: Job Queue.