Delayed events to GitHub

Incident Report for Buildkite

Resolved

Github have indicated they've resolve the networking issue and we have seen notifications latency return to normal.

Posted Jul 02, 2024 - 01:35 UTC

Monitoring

Github have rerouted traffic and we are seeing latency return to normal levels

Posted Jul 02, 2024 - 01:09 UTC

Update

A small percentage of notifications to Github (~10%) continue to be delayed by 30-600 seconds.

We're continuing to work with AWS and Github to identify and resolve this issue

The next update will be in 1 hour unless more information is available.

Posted Jul 01, 2024 - 23:41 UTC

Identified

Degraded network connectivity between one of our availability zones and GitHub is causing a subset of events to be delayed by up to several minutes.

Posted Jul 01, 2024 - 22:47 UTC

This incident affected: SCM Providers (GitHub) and Notifications (GitHub Commit Status Notifications).