We have seen no further errors in the past few hours, and this incident is now resolved.
Posted May 12, 2020 - 23:11 UTC
Monitoring
We've identified a problem with webhook ingestion on our load-balancer over the past few hours and have mitigated the problem. We will continue to monitor this closely. Lost webhooks can be retried by pushing a no-op commit to your source control or by starting a new build from the Buildkite dashboard.
Posted May 12, 2020 - 19:36 UTC
Investigating
We're receiving more reports that webhooks from source control systems are being lost in transit again and have started investigating the issue for other causes.
We recommend pushing no-op commits to run any lost webhooks again in order to trigger builds or to schedule manual builds.
Posted May 12, 2020 - 18:51 UTC
Monitoring
We've identified an unusual load pattern that caused errors in processing some webhooks. It has now passed, and we've also enforced limits to prevent it recurring.
Posted May 12, 2020 - 16:23 UTC
Investigating
We're investigating a reports of intermittent issues or delays processing incoming webhooks from source control systems over the past hour.
We recommend retrying affected builds while we investigate.