All Systems Operational

About This Site

Status updates for Buildkite’s services and components. You can also follow @buildkitestatus on Twitter for updates.

Uptime over the past 90 days. View historical uptime.
Web   Operational
90 days ago
99.97 % uptime
Today
Agent API   Operational
90 days ago
99.99 % uptime
Today
REST API   Operational
90 days ago
100.0 % uptime
Today
Job Queue   Operational
90 days ago
100.0 % uptime
Today
Notifications Operational
90 days ago
100.0 % uptime
Today
Github Commit Status Notifications   Operational
Email Notifications   Operational
Slack Notifications   Operational
Hipchat Notifications   Operational
Pusher Pusher REST API   Operational
Pusher WebSocket client API   Operational
Webhook Notifications   Operational
90 days ago
100.0 % uptime
Today
SCM Providers ? Operational
GitHub   Operational
Atlassian Bitbucket SSH   Operational
Atlassian Bitbucket Website and API   Operational
Atlassian Bitbucket Git via HTTPS   Operational
Third Party Services ? Operational
AWS ec2-us-east-1   Operational
AWS elasticache-us-east-1   Operational
AWS elb-us-east-1   Operational
AWS rds-us-east-1   Operational
Mandrill Mandrill Global   Operational
PagerDuty Notification Delivery   Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Database maintenance Jan 19, 10:00-11:00 AEDT
We'll be performing a minor upgrade on our database this weekend on Sunday 20th January between 11am-12pm UTC+11 (AEDT). We only expect a moment of actual downtime for instance cutover during this window. Agents and jobs should handle this transparently with retry, but the Buildkite web dashboard and APIs may be briefly unavailable.
Posted on Jan 13, 20:47 AEDT
System Metrics Month Week Day
Web Response Time ?
Fetching
Agent API Response Time ?
Fetching
REST API Response Time ?
Fetching
Agent Job Dispatch ?
Fetching
Error Responses ?
Fetching
Past Incidents
Jan 17, 2019

No incidents reported today.

Jan 16, 2019
Logs not loading
Resolved - The database appears to be stable again, and we aren't seeing further issues.
Jan 16, 07:46 AEDT
Update - We are continuing to monitor for any further issues.
Jan 16, 07:45 AEDT
Monitoring - One of our job log databases experienced a failover event to cope with underlying failures. During the failover some running and recent job logs and header times were not being stored or displayed correctly. Now the failover has completed any outstanding logs should have been retried by agents and stored correctly, and logs should be correctly displaying again. Please let us know if you see any further issues. We'll be keeping an eye on it.
Jan 16, 06:54 AEDT
Investigating - We're investigating an issue related to logs being unavailable for some users.
Jan 16, 06:36 AEDT
Jan 15, 2019

No incidents reported.

Jan 14, 2019
Database restart
Completed - Maintenance is complete, and everything is back to functioning normally.
Jan 14, 19:00 AEDT
Verifying - Reboot is completed, we're just checking everything's operating as expected.
Jan 14, 18:59 AEDT
Update - The reboot is in progress but is taking longer than expected, we'll be back soon.
Jan 14, 18:56 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 14, 18:45 AEDT
Scheduled - We will be restarting our database today, Mon 14th January, at 6:45pm AEDT (UTC+11), to relieve a Postgres bug which has presented in our main database. We only expect a moment of actual downtime for primary failover during this window. Agents and jobs should handle this transparently with retry, but the Buildkite web dashboard and APIs may be momentarily unavailable.
Jan 14, 12:27 AEDT
Jan 13, 2019

No incidents reported.

Jan 12, 2019

No incidents reported.

Jan 11, 2019

No incidents reported.

Jan 10, 2019

No incidents reported.

Jan 9, 2019

No incidents reported.

Jan 8, 2019
Intermittent notification delays
Resolved - We’ve kept an eye on our monitoring around this incident, and we believe we’ve caught all the regular culprits. Further mitigations for the root causes are being looked into. Thanks for your patience!
Jan 8, 16:12 AEDT
Monitoring - We’re keeping an eye on some further minor notification delays, related to the incident we reported resolved earlier. Apologies for our haste! Original incident: https://www.buildkitestatus.com/incidents/b9fmpb74ymwc
Jan 8, 14:24 AEDT
Intermittent notification delays
Resolved - We've tracked down the problem to a set of notifications that were causing high internal load in our background processor. We've disabled them and we're working on some alternatives to ensure this doesn't happen in the future.
Jan 8, 11:00 AEDT
Investigating - We've had some reports of intermittent delays with notifications (web hooks, GitHub commit statuses, etc). Our current systems aren't reporting any issues at the moment, but there seems to be a spike in delays every 12 hours or so. We're investigating the issue now.
Jan 8, 10:11 AEDT
Jan 7, 2019

No incidents reported.

Jan 6, 2019

No incidents reported.

Jan 5, 2019

No incidents reported.

Jan 4, 2019

No incidents reported.

Jan 3, 2019

No incidents reported.

← Incident History Powered by Statuspage