Oct 1, 06:57 UTC Resolved - The error rate has returned to normal levels. Oct 1, 06:33 UTC Monitoring - The mitigation is rolling out and we are seeing recovery. We will continue to monitor and provide an update. Oct 1, 06:10 UTC Identified - Since 2026-09-30 20:52:36 UTC on 30 September, small amount of incoming GitHub webhook deliveries to Buildkite pipelines are being rejected, so the related b…
Sep 30, 02:32 UTC Resolved - Outbound notification latency remains at expected levels. Some customers may have received duplicate notifications before the underlying error was resolved. Sep 30, 01:43 UTC Monitoring - Outbound notification queues have now fully caught up for all impacted customers. Sep 30, 01:19 UTC Identified - We've resolved the underlying error, and outbound notifications queues…
Sep 28, 01:30 UTC Resolved - Between 11:45 and 15:00 (GMT+10) on 28 September, macOS hosted agent jobs could not start because of a misconfigured agent image update. This has been fixed and new jobs are running normally. Jobs from that window still waiting in the Scheduled state won't start on their own. Please cancel and retry them.
Sep 22, 20:50 UTC Resolved - Services are fully recovered. Webhook ingestion was affected between 19:09 UTC - 19:32 UTC. Sep 22, 20:26 UTC Identified - Our engineers have addressed the issue and we're monitoring recovery. Sep 22, 20:07 UTC Investigating - We're seeing an issue affecting webhook ingestion and execution of GraphQL queries and are investigating the cause.
Sep 22, 00:32 UTC Resolved - We did not experience any customer impact during this AWS us-east-1 outage. Sep 21, 23:47 UTC Monitoring - We're aware of a major AWS outage in us-east-1, we are taking action to prevent impact and monitoring the situation.
Sep 21, 23:46 UTC Resolved - This incident has been resolved. Sep 21, 23:12 UTC Monitoring - We observed a slight increase in latency for build creation isolated to a few customers, this has now passed and we are seeing signs of improvement. Sep 21, 22:35 UTC Investigating - We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide…
Sep 10, 01:54 UTC Resolved - We have seen full recovery for customers since 20:28 UTC. We experienced an autoscaling feedback loop which increased the number of connections to our redis cluster above its ability to respond. This had widespread impact for all of our customers with Web UI, Agent API, REST API and job queue impact between 18:43-19:21 UTC, and again between 20:02-20:28 UTC. A full pos…
Sep 4, 02:39 UTC Resolved - Jobs running on Hosted Agents are now being dispatched promptly. Sep 4, 02:23 UTC Monitoring - We have deployed mitigations and have observed recovery across Hosted Agents. We will continue monitoring. Sep 4, 01:42 UTC Investigating - We're seeing increased latency and error rates within our Hosted Agents. We're currently investigating and will provide status updates as…
Sep 4, 01:40 UTC Investigating - We're seeing increased latency and error rates for a subset of our customers. We're currently investigating and will provide status updates as they become available.
Sep 3, 07:06 UTC Resolved - Jobs running on Hosted Agents are now being dispatched promptly. Sep 3, 06:44 UTC Monitoring - We're seeing recovery in hosted job scheduling rates. Your jobs should run, but there may be some delays as we work through any backlogs. Sep 3, 06:09 UTC Update - Some job dispatches to hosted agents are timing out, leading to jobs starting late. We're currently investigating…
Sep 2, 15:51 UTC Resolved - Job dispatch for Hosted Agents is no longer experiencing delays. Sep 2, 15:43 UTC Monitoring - We are continuing to process through the backlog of delayed jobs and monitoring. Sep 2, 15:29 UTC Identified - We're beginning to see recovery with delayed job dispatch for Hosted Agents. We are currently working through the backlog of delayed jobs. Sep 2, 15:16 UTC Update - O…
Sep 1, 16:52 UTC Resolved - Our outbound email queue has finished processing all previously failed deliveries. Email notifications are processing successfully. Sep 1, 16:38 UTC Monitoring - We are seeing successful email deliveries and are monitoring processing of our outbound email queue. Sep 1, 16:19 UTC Update - Our on-call engineers are working on failing over to an alternative upstream email…
Aug 27, 11:47 UTC Resolved - Processing of uploaded test execution data has caught up and is back to normal. Aug 27, 11:23 UTC Update - Catch-up processing continues, most of the backlog is processed, ingestion latency is decreasing. Aug 27, 10:56 UTC Monitoring - Internal processing of uploaded test execution data has been delayed, and is currently catching up. No data has been dropped, and all d…
Aug 26, 03:13 UTC Resolved - Mitigation strategies worked as expected and now agent registration is back to normal. All systems are stable now. Aug 26, 02:43 UTC Monitoring - Recovery is underway; we'll keep monitoring until recovery is complete. Aug 26, 02:36 UTC Update - We have rolled out a few changes to fix the underlying issues and are seeing signs of recovery.We'll keep monitoring until all…
Aug 26, 00:18 UTC Resolved - We're considering this incident resolved. Aug 25, 23:38 UTC Monitoring - Systems have recovered and we are seeing new builds created. For a period of about 20min new builds could not be created due to a failure of our internal DNS services. No build requests were lost during this time and we are continuing to process the backlog Aug 25, 23:23 UTC Update - We have ident…
Aug 25, 08:16 UTC Resolved - This incident has been resolved. Aug 25, 07:42 UTC Update - Agent connectivity is recovered, but some agents are not reconnecting automatically after the earlier forced disconnects.We recommend restarting agent service to make agents reconnect. Aug 25, 06:41 UTC Monitoring - At 06:04 UTC we deployed a defective request routing change to `agent-edge.buildkite.com`, whic…
Aug 17, 02:43 UTC Resolved - This incident has been resolved. Aug 17, 02:00 UTC Monitoring - We have confirmed that all background workers have caught up as of 1:13am UTC. Hosted Agents dispatch has returned to normal. We'll continue to to monitor the service and are checking remaining retries. Aug 17, 01:21 UTC Update - We’re investigating elevated latency in background worker processing between…
Aug 13, 16:40 UTC Resolved - We think the impact from the issue is over. Aug 13, 16:29 UTC Monitoring - We have identified and fixed the issue. We are monitoring and seeing signs of improvement. Aug 13, 16:14 UTC Identified - We've identified the cause of the issue and are actively mitigating. Aug 13, 15:41 UTC Investigating - We've spotted that something has gone wrong and there's elevated latenc…
Aug 12, 04:08 UTC Resolved - For the last week we have been internally testing our agent traffic running on our new EKS infrastructure successfully. At 23:01 11 August 2026 UTC we started routing
Aug 11, 02:04 UTC Resolved - This incident has been resolved. Aug 11, 01:04 UTC Update - The integration between Pipelines and Test Engine has been re-enabled, restoring the Tests tab on build pages. We’re monitoring the recovery to confirm the integration remains stable. Test results continue to be available directly through Test Engine. Aug 11, 00:50 UTC Update - The integration between Pipeline…
Aug 5, 22:00 UTC Resolved - Between 22:03 - 22:16 UTC customers experienced delayed job dispatch, build processing and notifications. Additionally there were errors receiving webhooks during this period.
Aug 3, 05:36 UTC Resolved - We've confirmed all impact from this incident has been resolved. Aug 3, 05:29 UTC Monitoring - The fix has been fully deployed and we are seeing signs of recovery. All events impacted by this issue will be automatically retried. We are continuing to monitor the fix for stability. Aug 3, 05:22 UTC Update - We've identified that the impact is more widespread that we initi…
Jul 28, 20:57 UTC Resolved - Notification latency for the affected shard is now under 5s. Jul 28, 20:38 UTC Identified - We are investigating delays to build and job notifications for a subset of customers.
Jun 18, 09:43 UTC Resolved - We have seen a full recovery of services. Jun 18, 09:17 UTC Monitoring - A fix has been deployed, services are recovering. Jun 18, 08:59 UTC Update - We've identified the issue, we're continuing working on rolling out a fix. Impact is restricted to the builds create API for a subset of tenants. Jun 18, 08:05 UTC Identified - We're seeing increased latency and error rat…
Jun 12, 00:16 UTC Resolved - The mitigation applied before the last update had the intended effect, and we have seen recovery in REST API latency. Jun 11, 23:51 UTC Update - We've isolated the issue to elevated load on our REST API service and have mitigated the issue. The agent and stacks API isn’t affected. Jun 11, 23:49 UTC Monitoring - We've isolated the issue to elevated load on our REST API…
Jun 11, 00:51 UTC Resolved - Between 00:05 - 00:34 UTC, a subset of customers experienced increased latency and timeout errors on the Agent API. This impacts job assignment. At peak impact, we saw an error rate of 1.3% of requests and job acceptance latency up to 53s. Jun 11, 00:32 UTC Investigating - We're observing increased latency and error rates for a subset of our customers on the Agent API.…
May 30, 00:30 UTC Resolved - We have received reports email deliveries have not been working, affecting signup and invite emails as well as build notification emails.This issue has now been resolved.
May 28, 21:18 UTC Resolved - This incident has been resolved. May 28, 20:47 UTC Identified - We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon. May 28, 20:20 UTC Investigating - We are investigating delays to build and job notifications for a subset of customers.
May 26, 10:38 UTC Resolved - We think the impact from the issue is over. May 26, 10:37 UTC Update - We see processing time for all affected services has returned to normal as of 20 minutes ago. May 26, 10:08 UTC Monitoring - We've identified the problem and have completed the remediation steps, we are now monitoring as service resumes. May 26, 09:56 UTC Identified - We're observing increased laten…
May 20, 17:39 UTC Resolved - The incident is resolved May 20, 17:26 UTC Monitoring - We are seeing recovery across affected customers and continue to monitor May 20, 17:06 UTC Identified - We have identified the issue and applied mitigations and are monitoring recoveryWe have determined that only a subset of customers are affected by the notification latency. May 20, 16:40 UTC Investigating - We a…