Incidents and maintenance reported on status page for Trigger.dev https://status.trigger.dev/ https://d1lppblt9t2x15.cloudfront.net/logos/bb72686f10eddb819a2958a98b50538b.png https://status.trigger.dev/ en https://status.trigger.dev/incident/1110434 Fri, 06 Nov 2026 12:27:22 -0000 https://status.trigger.dev/incident/1110434#3ce4b5c1223c153a93f78a9d6684b8563a1da79ad3fefeece8adfd0eefcde2ee Incident We’ve published our report on the November 1, 2026 security incident: [https://trigger.dev/blog/security-incident-report-nov-2026](https://trigger.dev/blog/security-incident-report-nov-2026) https://status.trigger.dev/incident/1110434 Sun, 01 Nov 2026 22:50:00 -0000 https://status.trigger.dev/incident/1110434#64cc312ab0e86e1301bb58f69697ac94d6343086a04f044bef81e7ecec7f2948 Incident There has been a security incident at Trigger.dev. We've blocked the exploit used in this attack but our investigation has identified exposure across a subset of organizations. The attacker gained unauthorized access to environment variables for these organizations. We have emailed all impacted account owners with steps to mitigate. We will follow up with more details. https://status.trigger.dev/incident/1083239 Tue, 06 Oct 2026 15:10:00 -0000 https://status.trigger.dev/incident/1083239#2d055b3cba683295731b7cef7b941b2142f1d89c02edce36b35e0cac177bc373 Incident We stopped the issue at 15:10 UTC and no environment variables have been cleared since. A permanent fix is in progress. Variables that were cleared between 12:21 and 15:10 UTC are still empty and will not come back on redeploy. We have contacted every affected customer directly. Re-entering them in the Trigger.dev dashboard under Environment Variables fixes the affected project right away. We're still working with affected customers to restore their values, so if you need help or have questions, reach out to us at support or reply to our email. https://status.trigger.dev/incident/1083239 Tue, 06 Oct 2026 12:21:00 -0000 https://status.trigger.dev/incident/1083239#f182f0593928b98b840f4763cb6649a3329289be680ec7a60ae8e02c66fe0084 Incident Between 12:21 and 15:09 UTC, deploying with the Vercel integration overwrote some environment variables with empty values in Trigger.dev. Projects with affected variables may see failing deploys or runs that error because of missing credentials. You are only affected if all of these are true: - Your project uses the Vercel integration with environment variable syncing enabled - The project has environment variables marked as Sensitive in Vercel - That project deployed between 12:21 and 15:09 UTC today If any of these don't apply, your project was not affected and you don't need to do anything. Only the environments that deployed in that window are affected. The cause has been fixed and no new variables are being cleared. Variables that were already cleared are still empty: redeploying will not restore them. We are working on restoring them, and we are contacting affected customers directly with the list of impacted projects and variables. If you can't wait, re-entering the values in the Trigger.dev dashboard under Environment Variables fixes the affected project right away. https://status.trigger.dev/incident/1076016 Mon, 28 Sep 2026 07:50:00 -0000 https://status.trigger.dev/incident/1076016#56ae7c70beefc5430ba8b634e95e62eb2109ab6c3e988787dc70da570c00b0f3 Incident Logs are being fully ingested again. The issue was with the internal workers in our shared OpenTelemetry service. They became unhealthy but weren't rolled. We're looking into why and how to prevent this from happening again. https://status.trigger.dev/incident/1076016 Mon, 28 Sep 2026 07:20:00 -0000 https://status.trigger.dev/incident/1076016#58d298e6a990b9ae92960ec4708bcfbbe301738a442219b5bb9be71aabcacdf3 Incident We're investigating run logs being dropped (from your run when you do console.log/logger.log). This doesn't impact run operation but does impact observability. This isn't due to a code or configuration change. More information soon. https://status.trigger.dev/incident/1049038 Fri, 04 Sep 2026 12:55:00 -0000 https://status.trigger.dev/incident/1049038#cde57f596e35bcf734ad7aa6d74317bfcd0878350ddb86d9a677e2bc4ac16cc9 Incident # Postmortem: Missing task logs and traces Date: 2026-09-03 Duration: ~67 minutes Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered ## Summary Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dropped. The events in those batches are permanently lost. Run execution, run status and the runs list were unaffected throughout, as were logs and traces that had already been stored. We are sorry for the missing data and for making affected runs harder to understand. ## What customers experienced Runs that were emitting logs or spans during the window may have a trace that is empty or missing spans, depending on which of their batches happened to land. The dashboard did not return errors: reads succeeded with fewer or zero rows, so this appeared as missing content rather than a failure. In a sample of dashboard trace queries during the window, about 41% returned no rows, versus about 2% normally. Separately, the OpenTelemetry ingestion endpoint returned some HTTP 500s and became slow, so tasks emitting a lot of telemetry may have logged export warnings. Runs outside the window were not affected. ## Root cause A release included two ClickHouse schema migrations run back to back on the task events table. The first dropped some unused indexes, which scheduled a background mutation over historical data. The second added a new input-only column. ClickHouse applies alters to a table in order, so the metadata change for the new column waited behind the index-drop mutation on the writer replicas. Our migration runner reported both as successful in under two seconds, because an ALTER returns once the replica it is connected to has accepted it, and our deploy pipeline treats that exit code as "the schema is ready". It does not check that the column exists on every writer replica or attempt a trial insert. The same release changed the task-event writer to include the new column in every insert, with no flag, fallback or separate schema-first release. Once the new code was running against replicas that did not yet serve the column, every task-event insert failed. The flush scheduler retries a failed batch three times and then drops it with no durable replay, which turned a temporary schema mismatch into permanent loss. We cancelled the background mutation and rolled the application back, but the failures stopped when ClickHouse Cloud autoscaled our writer replicas under memory pressure from the mutation, and the new replicas came up with the column visible. The rollback would have stopped the loss roughly twenty minutes after it was merged had that not happened. ## What we've done - Bounded the queues on the OpenTelemetry collectors that receive telemetry from task runners, in all regions, so a stalled database no longer causes them to run out of memory and return errors. - Re-rolled the release the following morning with the column visible on every replica, which went out cleanly. - Fixed a collector alert for dropped telemetry that had been configured in a way that meant it could never fire. ## Ongoing - Alert ownership and severity: routing alerts to the owner of the in-flight deploy, and making sustained production insert failures page. - A migration gate that checks capability rather than exit code: wait for outstanding mutations on the affected table to finish, confirm required state, and fail-closed. - A durable buffer for task-event batches that exhaust their retries, so a ClickHouse incompatibility or writer outage becomes a delay we can replay rather than a loss, with metrics that count what was actually dropped. - Ingestion lag monitoring: add alerts on how long it has been since a task event last landed in the database, and on how long merges and schema operations have been outstanding, so stalled or silently failing ingestion pages us regardless of the cause. We treated "the migration ran" as "the database is ready". The changes above are about making the release process prove that before a change goes out. We apologise for the missing logs and traces. https://status.trigger.dev/incident/1049038 Thu, 03 Sep 2026 16:20:00 -0000 https://status.trigger.dev/incident/1049038#2265f9bd7a72dab9879ad9be7119418babaafad2da54f873e722cf68ae02e5c3 Incident The ingestion delay is down to normal levels. A post-mortem will follow. https://status.trigger.dev/incident/1049038 Thu, 03 Sep 2026 15:50:00 -0000 https://status.trigger.dev/incident/1049038#9695b29e498973037ac598669dfc8f42461a5336ce6bc17a1db716afa3eb922f Incident Run logs/traces are inserting and the ingestion delays for are coming down. We have fixed the underlying issue. We will resolve this issue with more details when the ingestion delays are back to normal. https://status.trigger.dev/incident/1049038 Thu, 03 Sep 2026 15:28:00 -0000 https://status.trigger.dev/incident/1049038#b18a42453880e4f56554ff9d2127277050b7de6b06dce8c2e9e37917beda76b5 Incident We have identified the root cause of the issue and are working towards mitigating the issue. Currently the run list dashboard and API are fine, but we aren't able to insert new run logs and trace data. https://status.trigger.dev/incident/1049038 Thu, 03 Sep 2026 14:56:00 -0000 https://status.trigger.dev/incident/1049038#6859c8b857bd7eb7f42b6f500b82008cf07b7be1b172d2449d27b54f42fda3bc Incident The insert delay for the runs list and logs is longer than usual. This impacts the dashboard and APIs associated with listing runs and viewing run logs. This is being caused by higher load and slow auto-scaling in our ClickHouse cluster. https://status.trigger.dev/incident/1030875 Mon, 24 Aug 2026 15:34:00 -0000 https://status.trigger.dev/incident/1030875#bf4c0968d2d0fad90ea03125e867af6b6ab8ba92b6a655076e494906010c82ca Incident We are still working on the root cause patch and it will rollout as soon as its tested and ready. We're resolving this status update for now as the run list API and dashboard have been running in good health for the last 30 minutes https://status.trigger.dev/incident/1030875 Mon, 24 Aug 2026 14:56:00 -0000 https://status.trigger.dev/incident/1030875#562eab0cbf64c2781eca3066177a4b227a1fc817d1ebb1d21d6a2dbb35b802d0 Incident We have identified the root cause of the issue and are working on a fix. The runs list API and dashboard are currently healthy but the underlying root cause is still unpatched. https://status.trigger.dev/incident/1030875 Mon, 24 Aug 2026 14:25:00 -0000 https://status.trigger.dev/incident/1030875#245b191948f4d732972164b1aee56238d3ccdb0ac02f06e7fd76da015df26729 Incident The runs list dashboard and API is degraded leading to increased latency and/or timeouts.We're investigating and attempting to bring this back online. https://status.trigger.dev/incident/966232 Thu, 16 Jul 2026 16:26:00 -0000 https://status.trigger.dev/incident/966232#cdd3e4cd898f76f0c1a81240a8fa8042e6b187c4328b8f8631cc7695e69b842b Incident The dashboard is now available. If you are still experiencing issues please hard reload the dashboard application. https://status.trigger.dev/incident/966232 Thu, 16 Jul 2026 16:19:00 -0000 https://status.trigger.dev/incident/966232#fdd911032356f07da0e2452bbc34a94920af2cbdaa8eee279f8d16db8a8b4e79 Incident The dashboard is currently unavailable for all users. A fix has been identified and is being applied. Task runs are not impacted by this issue. https://status.trigger.dev/incident/966098 Thu, 16 Jul 2026 14:16:00 -0000 https://status.trigger.dev/incident/966098#6b2392e90b91297155568ae4a13c909eb0164dc7338e6623658c4efe34da492c Incident The issue is now resolved, the dashboard and API are operating normally. https://status.trigger.dev/incident/966098 Thu, 16 Jul 2026 13:49:00 -0000 https://status.trigger.dev/incident/966098#0413e3d2657bb3c9ec80715dc9cb8840a0fa7dc65331ab6b27cab526659407c9 Incident Runs list is recovering. Continuing to investigate and monitoring the situation. Run operations continue unimpacted. https://status.trigger.dev/incident/966098 Thu, 16 Jul 2026 13:27:00 -0000 https://status.trigger.dev/incident/966098#afcfb7de3a7160902539ed198fd3dc91a2f12394a20f5a6816c3bbbe3ac74011 Incident Listing runs on the dashboard and via the API is currently degraded. A fix is being rolled out now. Run operations are unaffected. https://status.trigger.dev/incident/965885 Thu, 16 Jul 2026 09:38:00 -0000 https://status.trigger.dev/incident/965885#3553d40c5ebc4eaba80e09f1d797152632d91823c69b88fff0e546c7f430b469 Incident SSO login access has been restored. https://status.trigger.dev/incident/964739 Wed, 15 Jul 2026 11:57:00 -0000 https://status.trigger.dev/incident/964739#6df725d65ead44bbbe23ed919558566cf56fcad0fad4cbd84be5232bf9f60682 Incident Root cause identified and a fix applied. We're provisioning additional ClickHouse resources to prevent this occurring again. https://status.trigger.dev/incident/939294 Wed, 01 Jul 2026 11:22:10 -0000 https://status.trigger.dev/incident/939294#376435e85d0d0ac88fd329f77b194c42ae0f8c55ad6a759fd25bfec00523e9ef Incident Update: the full postmortem is now published: https://trigger.dev/blog/incident-report-jun-30-2026 https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 15:54:00 -0000 https://status.trigger.dev/incident/939294#7660c21b1d22ed195715fa7f2f56bdbdc1ababcc68f534c442dc36de1e2308df Incident This incident has been resolved. We are continuing to monitor and a post-mortem will be posted soon. https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 15:43:00 -0000 https://status.trigger.dev/incident/939294#75a28b73c1f621b71a23e59f9aa0f78eaba9a43748fbf5a68d1dc04c71a5c5d8 Incident First, the important part: your runs are safe. Runs execute and are stored in Postgres as normal. This issue is isolated to ClickHouse, which we use as a read-optimised view store for querying runs. Triggering, execution, and storage have been unaffected throughout. This is a visibility issue, not a data loss one. The fix is downgrading ClickHouse to 25.12. That rollback runs at the ClickHouse Cloud infrastructure level, so we're coordinating with them to apply it. It's agreed and queued. ## What you're seeing While the view was degraded, the runs list, filtering, bulk actions, the query API, and all error monitoring and alerting were slow or unavailable, and dashboard charts, traces, logs, and span detail were blank or incomplete for affected runs. Run execution, billing, and your Postgres-stored run data were unaffected throughout. - Dashboard and API queries against run data are slow or failing. There was a window this morning where these reads were down entirely. - A small percentage of runs created since June 17 aren't yet in the correct state in the view. The runs ran and are stored. The view of them is catching up. - Run logs and spans have recovered. Run views are still catching up. ## What's causing it ClickHouse moved from 25.12 to 26.2 on June 17. The new version added a cap on JSON column complexity (`input_format_binary_max_type_complexity`, defaulting to 1000), measured across the combined outputs in each batched insert. Once a batch went over, those inserts stopped landing in the view. Then this morning, June 30, memory and CPU on the instance climbed sharply as the backlog of unmerged data caught up with it, driving reads down until it ran out of memory and caused today's outage. The 25.12 rollback resolves the root cause. ## What we've done so far - Cleared the data that couldn't merge so healthy merges can continue. Run logs and spans recovered quickly; run views are slower and still catching up. - Scaled the cluster horizontally to add headroom. - Lined up the 25.12 rollback with ClickHouse Cloud. https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 14:08:00 -0000 https://status.trigger.dev/incident/939294#8cef7813829b8c85677934a85706b5dbc4dffa90a9cad7cd544321c133bc8ad4 Incident We're continuing to monitor API response times, which are steadily improving. Our mitigation efforts remain ongoing. https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 13:17:00 -0000 https://status.trigger.dev/incident/939294#7d6a9e6372947f531eeee7eb1cbd37b2701c5c133d94219dd93cddf1c1b4cd0b Incident Runs are triggering and executing normally. The runs list (runs.list API + dashboard) performance is now improving. Run logs and spans are also improving. We're actively monitoring the rollout of the remediation. Next update in 30 mins. https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 12:36:00 -0000 https://status.trigger.dev/incident/939294#9d0be750bb4767904909d01af015a9b43ac2fcf62866207dd021f64d803eca62 Incident Runs are triggering and executing normally. The runs list (runs.list API + dashboard) is slow and may be delayed. Run logs and spans may be delayed. Cause identified, remediation in progress. Next update in 30 mins. https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 11:41:00 -0000 https://status.trigger.dev/incident/939294#c2189ee4c81a3b600a005cf04ca19dfcbbd9a5f2acacabcebfaa799eb62f73eb Incident The root cause has been identified and remediation work is taking place https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 11:06:00 -0000 https://status.trigger.dev/incident/939294#83db3c51b6002fcdb0da90413306693e0d71c94848f9b2841a1222192d735de4 Incident We believe slow ClickHouse replication is due to long running merges with deeply nested output objects. We have vertically scaled and are working with ClickHouse support on mitigations. https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 10:19:00 -0000 https://status.trigger.dev/incident/939294#3a226d0867b875ccaf1beb04405d2821048fd275ac76a2b6c571ad31d0214ec7 Incident Spans and logs are also degraded. This is due to slow replication into our ClickHouse cluster. We are investigating. https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 10:11:00 -0000 https://status.trigger.dev/incident/939294#8f52eee6da4d883100611996483bdd6795bdb2842ed1acd56a0babd424a28716 Incident The `runs.list()` SDK function and Runs dashboard page are degraded. This is due to slow replication into our ClickHouse cluster. We are investigating. https://status.trigger.dev/incident/939294 Tue, 30 Jun 2026 09:40:00 -0000 https://status.trigger.dev/incident/939294#e93c2c3a5eae1d8a79d76714e150b661157d1624b9b8627fd4e3609fbc6c41a7 Incident Runs are executing normally. The replication of runs to ClickHouse which powers the Runs list are impacted. We're investigating. https://status.trigger.dev/incident/931451 Sat, 27 Jun 2026 14:54:00 -0000 https://status.trigger.dev/incident/931451#3b0595bd2ac6f47955010e3b706735b6faa6faa13baaf76dfea13d5dda2d6e6b Incident The full postmortem has been posted here: https://trigger.dev/blog/incident-report-jun-22-2026 https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 21:43:00 -0000 https://status.trigger.dev/incident/931451#3efc32c920d9a73f4832625522da49fc10a1efac2427cd7e8f7205a0846f7690 Incident US East 1 and EU Central 1 dequeue performance issues have been resolved. We're continuing to monitor the situation and a full post-mortem will follow. https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 19:57:00 -0000 https://status.trigger.dev/incident/931451#5a8e9e343ec94869b93413a1abb7a61fb862ebe9816306da3bbe1502e59519ce Incident US East 1 dequeue performance degradation has cleared but we're continuing to monitor the situation https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 19:18:00 -0000 https://status.trigger.dev/incident/931451#61f296cf937cdf6923650fcd52f6dbd8580486af9f14839b5daadad45644e0f9 Incident US East 1 dequeues are currently degraded, as we work through the large backlog and deal with capacity issues with our upstream provider. https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 16:44:00 -0000 https://status.trigger.dev/incident/931451#ab4286b78c445393ec874f1bd5cd23509cd48da3c16733cbb79648b67ed4fbeb Incident US East 1 continues to operate normally. EU Central 1 scheduled jobs are also back to normal operation. The remaining impact is EU on-demand workers still catching up on the backlog, so you may see delayed dequeuing of queued runs for a little while longer. One thing to be aware of: if you have a large number of queued runs in EU Central 1, they can consume your concurrency and block runs in other regions (including US East 1) from executing. If your US East 1 runs aren't progressing despite that region being healthy, this is likely the cause. You can unblock yourself by cancelling the queued EU runs and replaying them in your normal region using the Bulk Action → Replay → Region override feature we just deployed. https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 14:29:00 -0000 https://status.trigger.dev/incident/931451#ed856a0478620dcb99b7ad7b409c8648937310ea1dc7fc18eea9f79f6eb7ebb0 Incident The EU Central 1 region continues to drain and it looks like will be back to normal operation within the next hour at current drain rates. We have deployed the Bulk Action -> Replay -> Region override feature now which allows replaying runs into a different region. US East 1 continues to operate normally. https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 14:00:00 -0000 https://status.trigger.dev/incident/931451#edb37bf2615cbbcf8beb03b3ba852a942c994900e896dd2aa2f0defafcf99d0a Incident Our EU Central 1 is back online and processing the backlog of runs in that region as fast as we can. We will update again when we have an expected clearance time to get back to normal dequeue operations. We're also working on adding the ability to the Bulk Action feature to be able to replay runs in a different region from the replayed runs. US East 1 continues to operate normally. Full postmortem to follow https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 11:29:00 -0000 https://status.trigger.dev/incident/931451#d2be3bc683dc3b7b49346f98390648c675dafbe976a784e9da6ef82434e54a49 Incident Our advice remains the same: switch your default region to us-east-1 if you can. More details below. **us-east-1** Operating normally with global queue times better than normal thresholds. If you have excessive runs queued in us-east-1: 1. Check your concurrency limits, increase as needed including at the account level from the Concurrency page (let us know via the in-app Help and Feedback and we will credit your account) 2. If you were on eu-central-1 they could be blocking your us-east-1 runs. Go to the Runs page and filter for "Queued" status and "eu-central-1" region. Then use the Bulk action on the right to cancel those runs. 3. When eu-central-1 is back you can replay these runs there. **eu-central-1** The cluster is being spun back up with more resources and the same settings we used to fix us-east-1. Currently no runs are executing here. The same thundering herd issue that happened in us-east-1 caused this issue as everyone moved their traffic over. This cluster was smaller so it took less to cause this problem. -- After this incident is over we will do a full post mortem with what happened along with a long list of things we need to do better. I've been noting them down and we've recorded this entire process. We will also be offering credits to all impacted customers. I know this doesn't cover the cost to your businesses. This is far from the bar we set for reliability. Sorry everyone, Matt https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 09:22:00 -0000 https://status.trigger.dev/incident/931451#de8ceb84bdcd22cd121224d57c4d474900b52b583a53291c96c20d3adf9b2e8f Incident us-east-1 is almost completely recovered although still with slower dequeue times than normal. We're continuing to investigate how to bring eu-central-1 back online. https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 08:47:00 -0000 https://status.trigger.dev/incident/931451#d0f67ff961880723b387911a15b7fcd14a263fd333738c6c910c795d234aefb5 Incident As our us-east-1 region is recovering the eu-central-1 region is now experiencing the same degradation that occurred last night to us-east-1. We're continuing to investigate and attempting to bring the dequeues back online for eu-central-1, while also working through the us-east-1 backlog, which is continuing to clear but won't be fully clear for 1+ hr at the current rate as we battle to get server capacity from our upstream provider. https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 07:19:00 -0000 https://status.trigger.dev/incident/931451#bfd2da7b7d15c268b2e3faa7f4d85dbe0cf0900475ea4b612a0c5a4e4e6ddb10 Incident We've scaled up our services to handle the increased workload, and throughput is improving. There's still a significant backlog of runs to work through, so you may continue to see delays while we catch up. We're monitoring closely and processing through the queue as fast as we safely can. EU Central 1 remains fully operational, so you can continue running workloads there in the meantime. https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 04:08:00 -0000 https://status.trigger.dev/incident/931451#25ebbe3719d770222065799e76ca30a38e3fff3266af9a50797d9fe1c16bda9c Incident We are now processing a high number of us-east-1 runs, but there is a large backlog to get through. The backlog is reducing and we're looking to see if we can do this faster in a controlled way. EU Central 1 remains fully operational, so you can continue running workloads there in the meantime. https://status.trigger.dev/incident/931451 Tue, 23 Jun 2026 01:30:00 -0000 https://status.trigger.dev/incident/931451#43f56d8569bbc04a9020db2c48a24c9cc117ae7e799cea7fad3283507f1bc82d Incident The control plane has stabilized and core services are coming back online. We’re carefully restoring capacity in a controlled way. EU Central 1 remains fully operational, so you can continue running workloads there in the meantime. No action needed on your end. We’re close to full resolution and will share our next update shortly. https://status.trigger.dev/incident/931451 Mon, 22 Jun 2026 23:20:00 -0000 https://status.trigger.dev/incident/931451#ba6e55ddddd4b38f69729efce5de3a1e99d40f2b11af870aecd4c11e7005e918 Incident For users who have switched from the us-east-1 region to the eu-central-1 region, and are seeing slow to minimal run executions for newly created runs, one way to unlock these new run executions is to cancel any of your us-east-1 queued runs, by using the Region filter in the runs list and selecting the Bulk Action -> Cancel option. This us-east-1 outage has discovered a bug in our concurrency accounting that can cause queued runs in degraded regions to be counted against your concurrency limit. We're continuing to make progress on restoring the us-east-1 region, and will update again shortly. https://status.trigger.dev/incident/931451 Mon, 22 Jun 2026 21:50:00 -0000 https://status.trigger.dev/incident/931451#8481ef9e16f2543132aabe9b577e1e2042e5a4183f35c45c899934ab9058e54d Incident Just a note about switching your default region to the eu-central-1 cluster, and the behavior of how the switching of regions effects existing queued runs: already enqueued runs keep their original region (which is set at trigger time). Switching your default region effects new runs only, immediately. It is possible to replay an individual runs on a different region, so one temporary workaround: - Cancel existing queued runs in the us-east-1 region (using the Region filter) - Go through each run cancelled individually and click through to the run detail page, and find the "Replay run" button in the top right - Select eu-central-1 from the drop down We're working on being able to streamline this process, and again we apologize for this outage. https://status.trigger.dev/incident/931451 Mon, 22 Jun 2026 21:37:00 -0000 https://status.trigger.dev/incident/931451#2e3a8fa61efb00389d50a1c18901f13f9eb539c145202b2eb9c9904dae3c13f6 Incident We are continuing to investigate the issue and with us-east-1 but have no hard timeline on when we might be able to recover. The root cause of the issue is still not clear to us, and the nature of the issue is making it very difficult to ascertain the underlying cause. We're focusing on mitigation. eu-central-1 continues to operate under normal conditions. https://status.trigger.dev/incident/931451 Mon, 22 Jun 2026 20:23:00 -0000 https://status.trigger.dev/incident/931451#65bde4a2e3ca9c016ece69745b0f14acc9ba75ac014bdd865046c477797450e4 Incident Runs have stopped executing in us-east-1, we're investigating. The eu-central-1 region is still executing runs, consider switching to that region as a temporary workaround. https://status.trigger.dev/incident/931451 Mon, 22 Jun 2026 19:48:00 -0000 https://status.trigger.dev/incident/931451#98f2ce8f8517fa4dd951d16e55433c28dfa78a9a1e48f3a78ade482a449fd282 Incident We managed to secure additional capacity, but large machines still disproportionately affected. https://status.trigger.dev/incident/931451 Mon, 22 Jun 2026 19:05:00 -0000 https://status.trigger.dev/incident/931451#908dc008d040ea8663c5e2c9a40451e71cc933c9f4a0aaf30a4876d2a3e9950b Incident This predominantly affects larger machine sizes. AWS is having capacity issues. https://status.trigger.dev/incident/903613 Sat, 23 May 2026 17:40:00 -0000 https://status.trigger.dev/incident/903613#9a69948de07ca65216578481572490a240346e11db605587f095ecf71b935f40 Incident This is now resolved, deploys should be working as normal. We've also implemented a change to prevent further DNS rollbacks. https://status.trigger.dev/incident/903613 Sat, 23 May 2026 15:00:00 -0000 https://status.trigger.dev/incident/903613#79c5b0367e238582d995be400a7f56afda720cbd7a82595f1602f609de033a84 Incident Trigger.dev deploy runs are failing with a `401 Unauthorized` push error. This is due to a downstream provider, we're working with them to get this back online. Workaround: deploy with --native-build-server: https://trigger.dev/changelog/deployments-with-native-builds https://status.trigger.dev/incident/902824 Fri, 22 May 2026 13:38:00 -0000 https://status.trigger.dev/incident/902824#1efeb5a5d54b3043dc050490fc74f7e452651ea5c1366d4cccc4072074b3b466 Incident This issue has now been resolved and services are operating normally https://status.trigger.dev/incident/902824 Fri, 22 May 2026 13:13:00 -0000 https://status.trigger.dev/incident/902824#4a7ef79e7bf66483612cd571f0457219fb56ddb05b4f11c5c15746164f9a0ce9 Incident Trigger.dev deploy runs are failing with a `401 Unauthorized` push error Workaround: deploy with --native-build-server: https://trigger.dev/changelog/deployments-with-native-builds https://status.trigger.dev/incident/876588 Thu, 23 Apr 2026 17:15:00 -0000 https://status.trigger.dev/incident/876588#cf3c2df37da702811418e96fee409b2bd7cc5f6162f3de35bf1b9f2324a6bf16 Incident ## Postmortem: Intermittent DNS Resolution Failures (US region) **Date**: 2026-04-21 **Duration**: ~1 hour **Impact**: Intermittent DNS resolution failures for tasks running in our US region ### Summary Between approximately 15:50 and 16:50 UTC on 2026-04-21, some tasks running in our US region experienced intermittent DNS resolution failures when connecting to external services such as databases, object storage, and third-party APIs. The errors typically surfaced as `EAI_AGAIN` or transient name-resolution timeouts. Our EU region was unaffected. The platform itself remained available throughout, and the incident self-resolved before any manual intervention. ### What customers experienced Tasks in the US region that made outbound connections to external hostnames during the affected window could see intermittent resolution failures. Cluster-internal communication and task scheduling were not impacted. We did not observe degradation of the dashboard, API, or engine. ### Root cause Under elevated workload, an internal telemetry service hit its memory limit and restarted. When it came back up, a large number of running tasks simultaneously re-established their connections to it, triggering a burst of DNS lookups for a single internal hostname. A default DNS configuration on our pods amplified that burst by roughly 10x at the wire level, because each hostname resolution attempted several search-path variants, and each of those was duplicated across the IPv4 and IPv6 query families. The telemetry service then went into a short restart loop, so the reconnection surge repeated, keeping DNS resolution degraded across the window. Some external name-resolution attempts from customer task code ran long enough to hit their own timeouts and surfaced as `EAI_AGAIN`. During the incident, our cluster DNS resolver remained healthy by its own metrics. The failure was silent at the node-network layer and was only fully reconstructable from after-the-fact packet capture. This is the main reason we could not attribute the degradation in real time, and it is the primary gap we have closed in the days since. ### What we've done - **Further scaled the telemetry service** horizontally, so there's more capacity headroom under workload spikes, and a simultaneous all-replicas restart is less likely. - **Deployed a per-node DNS cache** across our worker fleet to absorb redundant lookup patterns at source, before they reach the cluster DNS resolver. - **Reduced DNS amplification at source** on platform-managed services, so a single hostname lookup produces one wire query instead of ten. - **Extended our monitoring** for network-level silent drops, per-upstream DNS resolver latency, and telemetry pipeline health. - **Shipped a suite of production alerts** calibrated against this incident's actual signature. A future incident of this class should be detected in minutes rather than reconstructed after the fact. ### Ongoing - A separate observation about how our cluster DNS resolver forwards queries under load is under continued investigation. It is not currently known to cause user-facing impact, but it is a gap in our understanding that we are closing. - We are re-evaluating DNS defaults on customer-facing tasks to remove the amplification pattern entirely, rather than just absorbing it with caching. We take incidents of this class seriously even when they self-resolve. The multi-day investigation was a deliberate choice to reconstruct the event with enough fidelity to fix the underlying observability gap, not just the immediate symptoms. We apologise for the disruption. https://status.trigger.dev/incident/876588 Tue, 21 Apr 2026 16:15:00 -0000 https://status.trigger.dev/incident/876588#7166b7a1d89d010ab5dfceb3e9669e2c5ac93bb490a69277313e4a59d8c1bfc7 Incident The issue has been resolved and DNS service is back to normal. We'll continue investigation and share further details in this incident report. https://status.trigger.dev/incident/876588 Tue, 21 Apr 2026 15:40:00 -0000 https://status.trigger.dev/incident/876588#47494473672789070bc916d0e80790eb6da119af1ec3d09e7ab3fdcc60e30755 Incident We are seeing some DNS resolution issues in us-east-1. OTel spans in the web dashboard are also affected. We are investigating the root cause. In the meantime, please consider switching to the eu-central-1 region while we investigate. https://status.trigger.dev/incident/872348 Wed, 15 Apr 2026 14:40:00 -0000 https://status.trigger.dev/incident/872348#184b36730c74c71bbf2cc08843199dc8e3af080e7a1529dd869354ccd856fab7 Incident The issue is now resolved and DNS service is fully recovered. The degradation was caused by abnormal DNS load patterns, which caused intermittent issues with external resolvers. We've already applied some changes to harden our networking infrastructure and will monitor closely to confirm stability. https://status.trigger.dev/incident/872348 Wed, 15 Apr 2026 14:29:00 -0000 https://status.trigger.dev/incident/872348#119745c92ea5a32b1cd8f18fbd59c034f70cb6bdce3270579e4cd3f6deeece4c Incident This is a coredns issue. We have spread out the coredns pods onto more nodes and added more replicas. This wasn't the same issue as we have seen with previous DNS problems. We're determining why this happened still. We're monitoring the change as the error rate comes down. https://status.trigger.dev/incident/872348 Wed, 15 Apr 2026 14:03:00 -0000 https://status.trigger.dev/incident/872348#a418df41f6781e01233066a1089d443a465ee1e15c79efe88ca8c9580247d916 Incident This can impact HTTP requests including database queries. It seems to be impacting up to 5% of database queries for some users. https://status.trigger.dev/incident/850738 Tue, 17 Mar 2026 00:07:00 -0000 https://status.trigger.dev/incident/850738#636914f1e69b9b1b4b94408070eb3222955501a83d560727c80c4f61aa6ecf71 Incident DNS service is now back to fully operational. Increased traffic combined with a routine infrastructure rollout caused intermittent DNS resolution failures. We've tuned our DNS configuration to resolve the issue and are working on longer-term improvements to prevent recurrence. https://status.trigger.dev/incident/850738 Mon, 16 Mar 2026 21:20:00 -0000 https://status.trigger.dev/incident/850738#fbb82550c6a6b3553bc19833d0629fc5fa34c01e3dee2ba84de5bf2eb338d825 Incident From user runs we're seeing an increase in DNS related issues like: Error: getaddrinfo ENOTFOUND Error: getaddrinfo EAI_AGAIN We're investigating why this is happening. https://status.trigger.dev/incident/842530 Fri, 06 Mar 2026 15:15:00 -0000 https://status.trigger.dev/incident/842530#99cec9eb993e7fa0b46673f43e23526bf9a774ace7a2a20d0db7b316c81dd2b1 Incident The issue has been resolved. Dashboard and telemetry are now fully operational. https://status.trigger.dev/incident/842530 Fri, 06 Mar 2026 14:16:00 -0000 https://status.trigger.dev/incident/842530#292009857b826c8438e7a76ed6136b50a51e5a3395a89ef20ad6e5efc0d4d520 Incident The runs list and detail pages in the dashboard are currently degraded due to an ongoing issue with our ClickHouse DB. We're also observing some logs and span ingestion failures. We're currently investigating. Run executions are not impacted. https://status.trigger.dev/incident/836575 Sun, 01 Mar 2026 02:36:00 -0000 https://status.trigger.dev/incident/836575#b1f71e450fe2925b5f753f946651f4e38251cf930768e52d4d7c832163b2c105 Incident The issue is now resolved and dequeue times are back to normal. Mainly free-tier runs were affected. This was caused by a spike in the free-tier run volume. https://status.trigger.dev/incident/836575 Sun, 01 Mar 2026 02:28:00 -0000 https://status.trigger.dev/incident/836575#0b2e864a91be1250b31133cee326c5436320fab4b1124bfbd91b175f21fcb0b7 Incident We have identified the root cause and we're applying a fix. Mainly free-tier runs were affected. https://status.trigger.dev/incident/836575 Sun, 01 Mar 2026 01:39:00 -0000 https://status.trigger.dev/incident/836575#83b0ed212487b602464c88e7bd10a94dda135ab61ad3f3a381613fabe4a031e3 Incident Dequeues are slower than normal in us-east-1. Runs are still executing, but they are slower to start. We’re investigating the issue. https://status.trigger.dev/incident/810341 Fri, 23 Jan 2026 04:19:00 -0000 https://status.trigger.dev/incident/810341#dc4a827282a884337cf2a7cd65838e647463103b54352d465b7263a2f8c270bc Incident Full service has been restored. Task execution is back to normal. If you experienced failures between 01:37 and 04:19 UTC, those runs can be retried successfully now. What happened: During a period of high activity, a backlog of completed runs built up faster than our cleanup processes could handle, which put pressure on internal services and caused intermittent failures. What we did: We spun up additional cleanup capacity to clear the backlog and restore normal operation. What we're doing next: We're increasing resource limits on critical internal services and adding better alerting so we can catch this earlier if it happens again. https://status.trigger.dev/incident/810341 Fri, 23 Jan 2026 01:37:00 -0000 https://status.trigger.dev/incident/810341#d46991d6336051f9114f8b8e835c813ac9ab56d84432b6ad9ef9b13a5998c753 Incident We are experiencing intermittent issues that may cause some task runs to fail. Automatic retries are in place and should recover most affected runs. Our team is actively working on resolution. https://status.trigger.dev/incident/809042 Wed, 21 Jan 2026 17:32:00 -0000 https://status.trigger.dev/incident/809042#3d5656dafcc1532605352ca2e09c4de54bc1f057a49ab9da8ac14a2dd37d9408 Incident The root cause of the issue was a failed upgrade of our clickhouse service from 25.6 to 25.8. While the service failed to rollout, the rollback also failed and so our clickhouse service was split between some instances on 25.6 and 25.8, which caused various issues. Our clickhouse provider is continuing to investigate the failed release but we can confirm that the rollback has completed and queries are back to normal. https://status.trigger.dev/incident/809042 Wed, 21 Jan 2026 16:49:00 -0000 https://status.trigger.dev/incident/809042#bdf0e3a9d0f2fb1ab3ca2a55d26daf0d984a218a60af640eb539f88ec4e67f8a Incident The dashboard runs list is now live again but some v3 logs are being dropped because of ingestion problems. We're investigating and will provide an update as soon as we have anything https://status.trigger.dev/incident/809042 Wed, 21 Jan 2026 15:48:00 -0000 https://status.trigger.dev/incident/809042#315529cb405c961253198295ce3307d9f327487f6b7118eb8d7a83676b9cb121 Incident Our run sync to clickhouse process is currently delayed. The runs list in the dashboard will be behind but runs are executing as normal. https://status.trigger.dev/incident/808875 Wed, 21 Jan 2026 11:15:00 -0000 https://status.trigger.dev/incident/808875#9292b93e73516f8b0270447a6efc2c08fad3724cf6b5d2a6ff792d0fc328fa6a Incident All schedules have been fully restored. https://status.trigger.dev/incident/808875 Wed, 21 Jan 2026 10:56:00 -0000 https://status.trigger.dev/incident/808875#5d25e4f590f42351d0676c320865f6676e2f961dc81142d93fadd686b347cfb4 Incident We had a brief outage earlier which affected a subset of schedules. We are working on a fix to get them going again. https://status.trigger.dev/incident/805855 Fri, 16 Jan 2026 17:44:00 -0000 https://status.trigger.dev/incident/805855#b370768896d5ac3e4831cd5fa9c60ab01b4887fceb640cb583c0dc388d9d7b14 Incident We have finally been able to provision additional capacity and logs are working again. A full post-mortem will follow. https://status.trigger.dev/incident/805855 Fri, 16 Jan 2026 17:03:00 -0000 https://status.trigger.dev/incident/805855#73d6a10f66cabefff041b7ef72e4773ad431c4f72c111b87b0760015570b760c Incident Our task log storage system is currently overloaded and we are working on bringing up additional capacity, but in the meantime some logs may be lost. https://status.trigger.dev/incident/797559 Fri, 02 Jan 2026 21:16:00 -0000 https://status.trigger.dev/incident/797559#28871321d2c4ced4678f5dca66c1b33335b48093ccd716ed93b1c19d1b827d88 Incident The runs list is now up to do and syncing live updates again. https://status.trigger.dev/incident/797559 Fri, 02 Jan 2026 20:26:00 -0000 https://status.trigger.dev/incident/797559#6ea5959027eb18822f2081ff5897337ff99b241d8e7f3734f314b7692e51f00d Incident We've confirmed the fix has improved the situation but the runs list is still behind. It's currently just over 45 minutes behind but that's down from the max of 57 minutes. We estimate the runs list will be live again in about 1-2 hours. We're sorry for the delay and will be pushing fixing this sync service to the top of our priority in the new year. https://status.trigger.dev/incident/797559 Fri, 02 Jan 2026 19:53:00 -0000 https://status.trigger.dev/incident/797559#9d32ed9d1f4f0c34d3313ebc485e20b36d27fe39568746cf4f8e8e2b6dc7e365 Incident A fix has been deployed but the runs list continues to lag behind, but should be catching up soon. We're continuing to investigate and monitor the deployed fix. https://status.trigger.dev/incident/797559 Fri, 02 Jan 2026 18:51:00 -0000 https://status.trigger.dev/incident/797559#a11317c89e59581e85c569d3ccf4c00920750d6d9152256c236a77de7775efbe Incident We've identified the issue and are working on a fix now. Runs are still delayed (at the time of this update, the delay is about 32 minutes behind. https://status.trigger.dev/incident/797559 Fri, 02 Jan 2026 18:17:00 -0000 https://status.trigger.dev/incident/797559#251a11ed776746ad194490fa141e38c0f9f57f7ab5de7e2a4d35df320d91eef6 Incident Our run sync to clickhouse process is currently delayed. The runs list in the dashboard will be behind but runs are executing as normal. https://status.trigger.dev/incident/797058 Thu, 01 Jan 2026 15:10:00 -0000 https://status.trigger.dev/incident/797058#d19402994e51a18a82de25665f1a3db94cc544dbcbb5c152192f9654d4ee6474 Incident The new batch concurrency processing defaults have brought the processing queue down to zero https://status.trigger.dev/incident/797058 Thu, 01 Jan 2026 14:33:00 -0000 https://status.trigger.dev/incident/797058#508345570b920260e23f38d8a0b66c70864e87c41a4a2dcda48fa5bec09ae7ed Incident The batches are now processing much faster. The issue was the default batch processing concurrency for an organization was too low. Organizations that were already on Hobby or Pro had higher concurrency but newer orgs didn't. This created a backlog. We've increased the default processing concurrency for each org and will look at the upgrade path for paid customers as well. We're monitoring the processing – it should get back to near realtime soon. https://status.trigger.dev/incident/797058 Thu, 01 Jan 2026 10:00:00 -0000 https://status.trigger.dev/incident/797058#18c184c0e77ba336028bc2281a774c496238d48c77b038e2073f817c4e6907f2 Incident There is a backlog in processing `batchTrigger` and `batchTriggerAndWait` calls. This means runs are being created slower than normal for these. We're investigating why this is happening https://status.trigger.dev/incident/788344 Wed, 17 Dec 2025 15:30:00 -0000 https://status.trigger.dev/incident/788344#13c4e16d0ebb4e2cf2e3ff9cf4978b75ccfbae43d5d0f5f0322bc43c322e548d Incident Runs are now syncing live and the dashboard is back to normal. https://status.trigger.dev/incident/788344 Wed, 17 Dec 2025 15:12:00 -0000 https://status.trigger.dev/incident/788344#7712a74b0d146c6b75c7f846a01383da5df3335e6816730f842a17b96d5fa79b Incident Runs are not syncing to our clickhouse instances fast enough and so there is a delay in data in the runs list dashboard. Runs are operating normally. https://status.trigger.dev/incident/787788 Tue, 16 Dec 2025 22:42:00 -0000 https://status.trigger.dev/incident/787788#b30b3ad2ceb50586779a1ac98bfbaac54000b3704e6461f3a9e046d0b3f74400 Incident Operations have returned to normal, we're continuing to investigate the root cause and will provide more detail as we know more. https://status.trigger.dev/incident/787788 Tue, 16 Dec 2025 21:02:00 -0000 https://status.trigger.dev/incident/787788#48afc55a8d8d4fa57916b7f9133a2475fb36ed9011c693281a9801cbbbf91592 Incident We’re seeing a percentage of queries failing from ClickHouse Cloud which powers some pages in the dashboard, like Tasks graphs, Runs page and the logs. We’re talking to their team to try resolve this. https://status.trigger.dev/incident/787481 Tue, 16 Dec 2025 11:05:00 -0000 https://status.trigger.dev/incident/787481#32da1cde4666a3187009a963332938e10f8e141e1e8c9d2a3f74aa88ff9b3e78 Incident The issue in ClickHouse is now resolved. The dashboard is back to being fully operational. The root cause was a faulty node in the ClickHouse cluster which we couldn't kill. We're speaking to the ClickHouse Cloud team to find out why it happened. https://status.trigger.dev/incident/787481 Tue, 16 Dec 2025 10:43:00 -0000 https://status.trigger.dev/incident/787481#49f493b9827d3635505d4ed96902ec2d12c1ede81087900bc32862a2f4348fa7 Incident The dashboard is currently degraded due to an ongoing issue with our ClickHouse DB. We're currently investigating further. Run executions are not impacted. https://status.trigger.dev/incident/780932 Fri, 05 Dec 2025 18:52:00 -0000 https://status.trigger.dev/incident/780932#66fad50dc79ce3a177cc00582469e9409dc2b6ea566ef8c08666b8ada84c0aee Incident Deployment for new projects have been fully restored. https://status.trigger.dev/incident/780932