Knock is an agent-native customer engagement platform. Knock helps teams power their customer messaging for transactional, lifecycle, and marketing in a single tool across in and out-of-app channels like email, push, SMS, and more.

In mid 2026, the platform engineering team at Knock undertook a project to tackle infrastructure spend and optimize margins by reducing COGS (Cost of Goods Sold). As part of this project, we decided to tackle a long-standing source of headaches for us: our “data streams” built on Amazon Kinesis.

By replacing our Kinesis infrastructure with WarpStream, we expect to reduce annual data streaming costs by 77%, from about $112,000 to $25,000. The migration took one engineer about a month and moved five topics and roughly 20 consumers without service disruption.

The limits of Knock’s Kinesis architecture

As a messaging service, Knock produces and processes a lot of data — observability logs, engagement events, and product events that drive messaging flows. And because we handle customer data, teams need to trust Knock with their privacy, which brings another set of challenges.

Knock’s backend services are primarily written in Elixir. At the very beginning of the company’s development, we picked Kinesis as our “message hub” for a variety of events broadcast to the rest of the system — events related to the lifecycle of messages, state updates from our in-house workflow engine, and more.

Kinesis is a reliable technology, but it proved to be a costly operational burden for us. Kinesis is designed to be used by KCL (Kinesis Client Library), the Java driver built by AWS itself. That driver is complex. Being on Elixir, we had to instead rely on a community driver that caused us many issues over the years.

The lack of support for enhanced fan-out in our driver meant we had to concoct a “manifold” abstraction for fanning events out to multiple consumers. A manifold was made of a parent Kinesis stream consumed by a “fan-out worker”, whose job would be to republish each message to a set of children Kinesis streams. This worked just fine for a few years, but it caused occasional inconsistencies and was a major source of wasted infrastructure spend — we’d pay for the bytes we put on the parent plus each child stream.

Kinesis alternatives and picking WarpStream

A few months ago we started looking around for solutions. We researched message broker solutions (like RabbitMQ), native AWS solutions (like SNS topics fanning out to SQS queues), newer technologies like Apache Pulsar®, and event logs like Apache Kafka®.

Our problem space is event-log shaped: we push too much data on the wire for solutions like RabbitMQ (copy on each queue) or SNS/SQS, which turned out to be prohibitively expensive. Kafka has good support in the Erlang / Elixir ecosystem via the brod driver and the broadway_kafka integration with Broadway, so we pretty quickly landed on it. However, we were concerned by the cost of AWS MSK and self-hosting was not an option.

WarpStream to the rescue. WarpStream provided Knock with all the things we were looking for. Its higher latency — in the standard, cost-effective configuration — was not a problem for our use cases. We were able to replace “manifolds” and their children with Kafka-native topics and consumer groups. We now operate topics that range from two to eight different consumers each, where adding a consumer doesn’t require a new stream (or significant additional costs).

WarpStream (by way of Kafka semantics) also unlocked fan-in for us: we can now easily have consumer groups that consume from multiple topics, something we had to hack together on Kinesis.

WarpStream’s BYOC model also simplified our privacy and compliance review. Unlike a managed broker that stores customer data in the provider’s cloud account, WarpStream keeps our message data in S3 inside our own AWS accounts.

Migrating to WarpStream without service disruptions

The whole migration went really smoothly. We de-risked the cutover by running the same consumer code against both Kinesis and WarpStream at the same time. Then, we transitioned published events from one to the other via extensive feature flagging. We publish plenty of metrics, so we were able to observe traffic shifting over to WarpStream topics.

The whole migration (five topics and around 20 consumers in total) took us about a month with most of the work carried out by one engineer.

Operating WarpStream at Knock

WarpStream turned out to be exactly what we were looking for. We’ve been operating it for a few months now, and it met our cost, reliability, and operational goals. We’re able to run our main application’s needs off of three m6in.xlarge EC2 boxes, which sit at a comfortable ~20% CPU usage. By reserving those instances as well, we’re paying an extremely cost-effective ~$400/mo for all those instances.

We haven’t had the need for horizontal autoscaling yet, but we’re feeling great about the fact we could just throw more nodes at our main Agent group if our usage ramped up significantly. We also run two additional, lower-cost agents for ClickPipes (see below).

We were also pleasantly surprised by the latency behavior we’ve observed. None of our use cases needed sub-second end-to-end latency; we observed a consistent sub-2s latency — less than we expected and good enough for our use cases, which often are centered around eventually-consistent logs.

As an aside, we were extremely happy with the number and quality of metrics exposed by WarpStream. Our main observability platform is Datadog, where WarpStream is a supported integration: this allowed us to enable high-cardinality metrics (per topic and per partition!) without incurring any additional custom-metric-related costs.

Knock is an avid ClickHouse user. In fact, all our Kinesis manifolds had one consumer each dedicated to persisting data from the stream to ClickHouse. Just a couple of months before the WarpStream migration, we had started making heavy use of ClickPipes, ClickHouse Cloud’s solution for natively ingesting data into ClickHouse.

The fact that WarpStream is an officially-supported source for ClickPipes was an important factor in our decision to use WarpStream. We now run a ClickPipe attached to each WarpStream topic. We spun up a dedicated Agent group for ClickPipes to isolate their traffic patterns. We loved being able to constrain that group to only serving Fetch requests (meaning cheap nodes).

Cutting Kinesis costs

Our production traffic on WarpStream sits at around 10–15k messages per second, or around 15 MiB/s of uncompressed data published to our topics.

We expect to reduce our yearly spend significantly. Kinesis was costing us approximately $112k/year between shard-hours and DynamoDB tables for checkpointing. We’re (painfully!) aware that our custom manifold design made Kinesis more expensive than a more-idiomatic enhanced fan-out setup. However, our estimates tell us that even an optimal Kinesis setup would cost us more than the roughly $25k/year we’re forecasting with WarpStream — a whopping ~77% decrease from today. This figure includes:

• AWS S3 costs.
• WarpStream control plane (Pro cluster tier).
• AWS EC2 nodes for the Agent groups (five total nodes, reserved instances).

This doesn’t count ClickPipes costs as we were paying those exact same costs with Kinesis.

For us, an important improvement here is the agility we gained. Adding a consumer to an existing topic costs us very little (only the additional Fetch requests); that’s something we do consistently. Adding topics, or rather adding data throughput, costs money but it’s cheaper than it was on Kinesis.

What changed for Knock

The main result of the migration is undoubtedly COGS savings. However, we ended up with quite a few additional positive outcomes:

• We were able to ditch a problematic driver that we were essentially maintaining in house at this point. Less code, fewer problems. WarpStream — and the Kafka driver we’re using — have proven to be quite reliable so far.
• Our system became more elastic. Scaling up is required less often (no more frequent resharding!), and when needed, it consists of just adding Agent nodes to our Agent groups.
• We gained the ability to fan-in from multiple topics to one consumer group.

The end result was not only a cheaper system, but also a simpler system overall. We removed infrastructure that caused repeated problems for us, turned adding consumer groups to topics into a non-event, and gave the team a solid Kafka-compatible foundation with great Elixir support.

When we built WarpStream, the goal was to redesign Apache Kafka® from the ground up in a way that was dramatically cheaper and easier to operate: stateless Agents, object storage, and no disks to manage or partitions to rebalance. But there has always been a catch: before you can run Kafka the easy way, you have to get there, and migrating an existing Kafka cluster to WarpStream can be notoriously difficult. A cluster rarely serves a single application, many teams produce and consume through it, and all of them have to be migrated with minimal disruption to production traffic.

At a high level, that work breaks down into three parts: migrating the data, migrating the consumers, and migrating the producers. The data has to be replicated without loss or reordering. Consumers need their group offsets carried over so they can resume where they left off. And producers, the hardest of the three, need to be moved quickly and with minimal downtime, since pausing them means halting durable writes, with real potential for data loss if anything goes wrong.

What Orbit Already Solves

Part of this picture is already solved by Orbit, WarpStream’s built-in replication feature, which makes migrating the data and the consumers easy. Orbit continuously replicates topic data and consumer group offsets from any source Kafka cluster into WarpStream while preserving the exact offset of every record.

This makes migrating consumers straightforward: since the data and committed offsets are replicated, Kafka consumers can simply switch their bootstrap URL from the source cluster to WarpStream and pick up exactly where they left off. There is no reprocessing, no skipped records, and no shared cutover window, because each consumer group can move whenever its team is ready.

Migrating Producers Was Still Hard

That leaves the producers. Until now, migrating them was an entirely manual four-step process. It involved:

1. Stopping every producer for a topic.
2. Waiting for Orbit’s replication lag to reach zero.
3. Disabling Orbit replication for the affected topics.
4. Restarting the producers with their bootstrap URLs pointing at WarpStream.

In practice, this usually translated to downtime and an operator watching a dashboard and deciding when it’s safe to migrate. Worse still, because a single forgotten producer still writing to the source cluster means data loss, every team had to move together. Producers were the one part of migrating to WarpStream that still required a maintenance window.

Some of our most sophisticated customers that couldn’t tolerate any migration downtime wrote custom scripts and modified their application code to leverage dynamic feature flags to automate the migration as much as possible. Some even used clever tricks like applying ACLs on the source cluster as an automated “signal” to their application that it was time to instantiate a new client and cut over to WarpStream.

We wanted to solve this problem once and for all for our customers, but many of the automated solutions that our customers came up with weren’t practical solutions for us. We couldn’t rewrite each of our individual customer’s applications to use dynamic feature flags, for example.

Another potential solution was for us to provide our customers with custom WarpStream “fat clients” that they could embed in their applications that would automate the migration process for them, but that had its own challenges as well. The clients would have to work in every programming language, and for many of our enterprise customers getting many different teams to swap out the clients in their application was an impossible task.

As a result, we settled on a different approach: a proxy. Or more specifically, a very limited type of proxy that only does one thing: forward produce requests. We reasoned that customers could update the bootstrap URL for all of their applications to point to the WarpStream cluster, and then WarpStream would just forward those Produce requests to the source cluster. This way, every team could move at its own pace to update their application with zero downtime, but also it would allow us to concentrate all of the Produce traffic in a single location that we controlled so that we could automate steps 1 through 4 from above for our customers.

The result is Orbit Auto Migration: seamless migrations of Kafka producers from any source cluster to WarpStream with zero downtime, zero restarts, and zero maintenance windows. Producers update their bootstrap URL to point at the WarpStream cluster ahead of time, at whatever pace suits each team, and the WarpStream Agents transparently forward those Produce requests to the source Kafka cluster.

When an operator decides to migrate a topic, they click a button in the WarpStream UI and then the following steps happen automatically:

1. The WarpStream Agents start blocking traffic from the Producer clients. This is fine because it lasts for only a few seconds and the Producers will buffer and retry.
2. The Agents wait for topic and offset replication lag to reach zero.
3. The Agents sever the Orbit replication link.
4. The Agents stop rejecting traffic from the Producer clients. The topic is fully migrated.

Automating the old manual process removes the downtime and coordination, but that alone does not make a migration seamless. The cutover itself still has to be safe to attempt and easy to unwind if something goes wrong, producers still have to be ready for a cluster that behaves differently than the one they were tuned for, and the migration has to stay visible throughout.

So, against this backdrop, how do you cut over thousands of producers without stopping any of them, without asking every team to move at once, and with a way to watch, abort, and roll back the migration at any point? The easiest way to explain it is to follow one topic through its migration journey, from start to finish.

The Life of a Migration

At a high level, Auto Migration has two main components. In WarpStream’s control plane, a migration controller runs a state machine for each topic and decides when it’s safe to move from one stage of the migration to the next. In the data plane, every WarpStream Agent contains a proxy that can forward produce requests to the source cluster. The Agents continuously watch each topic’s state and adjust how each produce request is handled accordingly.

Each topic’s state machine moves through four states: REJECT, PROXY, MIGRATING, and COMPLETE.

Every topic that Orbit replicates starts out in the REJECT state, where WarpStream refuses produce requests to it. While Orbit is copying records from the source, WarpStream’s copy of the topic has to match the source record for record, offset for offset. A write from anywhere else would break that, so a producer that attempts to produce directly to the topic in WarpStream simply gets an error.

When Auto Migration is enabled for a topic, it moves to PROXY. This is the state in which producer applications actually update their bootstrap URLs to point to WarpStream whenever they’re ready. The Agents see that the topic is in PROXY and forward every produce request to the source cluster. Orbit, meanwhile, keeps replicating the topic in the background from the source into WarpStream, bounding the replication lag between the two clusters. Because WarpStream Agents are stateless, any Agent can forward requests for any producer, and there is no dedicated proxy tier to size, scale, or fail over. A topic can sit in PROXY for as long as teams need, whether that is an afternoon or several months.

Once every producer has updated its bootstrap URL, the cutover can begin. When a cutover is initiated, the control plane first checks that replication lag is low; if WarpStream’s copy of the topic is too far behind, the migration does not start. Otherwise the topic enters the MIGRATING state. The Agents pick up the state change, stop forwarding writes, and respond to every produce request for the topic with a retriable error. Producers end up in a brief holding pattern, holding their writes in client buffers and retrying automatically until the cutover completes and their writes go through, with nothing surfacing to the application.

While the producers retry, the control plane finalizes the cutover. It waits for every Agent to confirm that it has stopped forwarding writes for the topic and that any requests still in flight to the source cluster have finished. It then waits for Orbit to drain the remaining replication lag to zero, and verifies that every record the source cluster acknowledged now exists in WarpStream. Only when all of this has happened does the topic move to the COMPLETE state. The producers’ next retry succeeds, and this time WarpStream serves the write itself. The whole window typically lasts on the order of seconds.

If the lag does not drain within the configured migration timeout, the topic automatically rolls back to the PROXY state. The Agents resume forwarding to the source, producers carry on as before, and the cutover can be retried later. A migration can also be aborted manually at any point before it completes.

Once a topic reaches COMPLETE, the migration is finished and the topic behaves like any other WarpStream topic. WarpStream serves its writes directly, Orbit stops managing it, and the proxy is no longer involved. Throughout all of this, no producer was ever stopped or restarted, and no data was lost.

No Surprises

A migration is a one-way door: once a topic has cut over and its producers are writing to WarpStream, there is no easy way back. Everything that could threaten the cutover like a mistuned producer, a missing ACL, or a forgotten writer, needs to surface as early as possible, and the process itself needs to stay visible and auditable throughout.

To truly make migrations stress-free for our customers, Auto Migration had to address each of these concerns as well, not just the mechanics of the cutover.

One potential surprise at cutover would be latency. WarpStream writes directly to object storage, so a produce request takes longer to acknowledge than it does on a disk-based Kafka cluster. Sustaining the same throughput at a higher latency means keeping more data in flight, which in practice means larger batches, more in-flight requests, and bigger buffers for records that are waiting to be acknowledged. The relationship is roughly max throughput ≈ (in-flight requests × request size) / acknowledgement latency, as per Little’s law. A producer tuned for single-digit millisecond responses often has its queues and buffers sized too small for this, and when acknowledgements suddenly take hundreds of milliseconds, its maximum throughput can fall below the rate at which the application writes. Records enter the buffer faster than they leave it, causing it to fill, and the producer stalls or starts failing writes.

This is especially dangerous during a live migration, because producer settings are fixed at startup. The same configuration has to work both while the topic is proxied and after the cutover.

If proxied requests were serviced at the source cluster’s latency, an untuned producer would behave perfectly through the entire proxy phase and only start failing after the cutover in the higher latency regime, exactly when there is no easy way to revert the migration.

To handle this, the proxy makes the latency producers’ experience in PROXY similar to what they would see producing to WarpStream directly. For every proxied produce request, it estimates how long WarpStream itself would have taken to durably store the batch, accounting for the time spent batching, writing to object storage, and committing metadata, and it holds the response until that much time has passed. Producers still have to be tuned for WarpStream, with larger batches, more requests in flight, and bigger buffers. The difference is now teams can tune their producers early, during the proxy phase, with data still flowing safely to the source, instead of firefighting after the cutover. And because producers end up tuned in advance, latency barely changes at cutover, resulting in a smoother migration. This has already paid off in practice; it caught a misconfigured producer in one customer’s migration well before their cutover.

Authentication and authorization follow a similar pattern. The proxy enforces both on every request i.e., a producer has to present valid WarpStream credentials and pass WarpStream’s authorization checks before its write is forwarded, even though that write is ultimately served by the source cluster. A missing credential or an incomplete ACL therefore surfaces the first time a producer connects, rather than after the topic has migrated.

Minimizing Drain Time

Ensuring that the pre- and post-migration regimes match removes one class of risk but the cutover itself carries another. While a topic is cutting over, its producers are retrying, which means their writes sit in client buffers and nothing is being stored durably. If a cutover were allowed to start while replication lag was high, that window could stretch from seconds into minutes, long enough to fill producer buffers and, in the worst case, run producers out of memory and drop data. So the control plane only starts a cutover when it is likely to finish quickly. When a migration is initiated, it checks the topic’s replication lag and refuses to begin if it is too high. The check uses time lag rather than offset lag, because offset lag says little about how long a drain will take. A backlog of a million records might be a few seconds of production on a high-throughput topic or a week of production on a low-throughput one.

Time lag measures how far behind the copy is in time, and it translates directly into drain time, since a topic running ten seconds behind the source will drain in roughly ten seconds once writes are blocked. Anything that would not drain comfortably within the migration timeout is rejected up front, before its producers are affected at all. Together with the automatic rollback described earlier, this protects the cutover at both ends, as a migration only begins when it is likely to succeed and gracefully rolls back on its own when it does not.

The migration window itself needs to stay short for the same reason. Every moment it lasts is a moment producer writes wait in client buffers instead of being stored durably, and its length is decided almost entirely by how quickly Orbit drains the remaining lag. So we made the replication scheduler migration-aware. Topics that are cutting over are prioritized in the replication queue, ahead of everything else Orbit is copying. Topics migrate independently, so when several are cutting over at once, the one with the least remaining lag drains first (shortest-job-first), which minimizes the average time a topic spends in MIGRATING. Within a topic, the objective flips because a topic is only done when every one of its partitions has drained. So the partitions with the most lag drain first (longest-processing-time-first), giving the partition that decides the finish time a head start. Together, these keep the time producers spend retrying as short as possible.

Detecting Rogue Clients

Everything so far assumes that every producer for a topic has actually been re-pointed before its cutover begins. In practice, producers are scattered across teams and services, and often nobody has a complete list of where they all live. A forgotten producer that is still actively writing to the source cluster keeps the replication lag from ever draining to zero, leaving the topic’s migration stuck in a loop of timeouts and rollbacks.

To make it easier to detect these forgotten producers, we made Orbit proxy-aware. The proxy attaches a small internal header to every record it forwards, and because Orbit sees every record on the source topic as it replicates, it can tell which records came through the proxy and which were written directly. If unproxied records keep appearing on a topic, WarpStream raises a diagnostic that names the topic, and the forgotten producer is found while there is still time to move it.

A more subtle case is a producer that writes only occasionally, like a cron job that runs every twelve hours. Because it produces nothing while the topic is proxied, there are no unproxied records for Orbit to flag, and because it produces nothing during the cutover, the replication lag drains normally and the migration completes cleanly. Then the cron job fires, and its records land on a source cluster that nothing is replicating anymore. To catch this, WarpStream snapshots the source topic’s end offset at the moment the migration completes, and for the next 24 hours it keeps checking the source against that snapshot. If the offset ever advances, a diagnostic fires naming the topic. Catching a producer at that point is admittedly late, since the records it wrote are sitting on the wrong cluster, but an alert beats silently losing data, notifying the team to take action.

The same visibility extends to the migration itself. An event is emitted every time a migration is initiated, completes, times out, fails, or is aborted, so the full history of who started what, when, and how many attempts it took is written as it happens rather than pieced together afterwards.

Correctness at Scale

Making all of this work correctly, and keeping it performant at the scale of real production clusters, meant solving a number of technical challenges.

One challenge was how to forward produce requests to the source cluster. A single request can carry batches for many partitions, and each of those partitions has its own leader among the source cluster’s brokers.

The obvious approach is to hand the records to an off-the-shelf Kafka producer client pointed at the source, but a producer client does not pass records along untouched. It re-partitions, re-batches, and re-compresses them, retries them on its own schedule, and stamps them with its own producer identity. What arrives at the source is no longer the request the original producer made, and so the proxy has no way to build the exact response the producer is waiting for. Instead, the proxy handles every step of the forwarding itself, working directly with the raw requests and record batches. It keeps a cache of the source cluster’s metadata, splits each incoming request by partition leader, forwards the producer’s original batches to the right brokers in parallel, and reassembles the responses into a single answer for the client.

Another was knowing when it is actually safe to complete a cutover. Completing a migration too quickly can result in data loss. For example, if a record reaches the source cluster and the producer received an ack but the cutover completes before it has been replicated, that record is left behind, stranded on the source that Orbit is no longer replicating.

During the MIGRATING state, the control plane waits for the remaining replication lag to drain to zero, switches replication off for the topic, and lets WarpStream serve its writes directly. To avoid data loss, replication can only be switched off safely once every in-flight proxied write has either completed and been replicated, or been blocked, because a record that reaches the source afterwards will be stranded, as described above. The difficulty is that Agents learn about the state change asynchronously, so for a short while some Agents have stopped forwarding while others have not, and an Agent that is partitioned might hold a proxied write in flight far longer than expected.

Before completing a cutover, the control plane therefore waits for every Agent to confirm two things: (1) that it has started blocking writes for the topic, and (2) that every proxied request it had in flight has finished. Each Agent tracks its in-flight proxied requests and reports the topics it has blocked and drained through its regular heartbeats. The control plane does not proceed until every Kafka-serving Agent has confirmed both for the current migration attempt; any Agent that cannot report its state blocks the cutover.

All of this works for any Agent the control plane can hear from. An Agent that has stopped responding altogether reports nothing at all, and so that case must be handled separately.

That case is handled by the way an Agent learns a topic’s state in the first place. Each Agent reads migration state through a local cache that it refreshes from the control plane every couple of seconds, and two rules govern how that cache may be used. An Agent will not forward a write based on a cache entry more than a couple of seconds old, and a forwarded write that does not complete within a few seconds of the cache read that admitted it is answered with a retriable error rather than a success, even if it later lands on the source. In effect, an Agent’s permission to proxy a topic is a lease that expires within seconds, and the only way to renew it is to fetch the topic’s current state. A partitioned Agent cannot renew, so it stops forwarding on its own, and anything it still has in flight is answered with a retriable error when its window runs out. An Agent that recovers renews its lease only by learning that the topic is now migrating, so it comes back fencing writes rather than forwarding them. Once a topic enters MIGRATING, every Agent that refreshes its cache starts blocking requests at once, and any Agent that cannot refresh loses the ability to forward within seconds as its lease runs out. The control plane waits out that window, after which no successful proxied write can exist anywhere, and only then measures the lag, drains it, and switches replication off. This guarantees that every write acknowledged to a producer reached the source while replication was still running, and was therefore replicated into WarpStream before the topic switched over. No acknowledged record is ever left behind.

There is one more twist. When a proxied request times out, the Agent has no way to know whether the write actually landed on the source cluster. The only safe answer is a retriable error, since telling the producer its write succeeded when it may not have is exactly the loss we set out to prevent. The producer retries, and if the original write did land, the topic ends up with a duplicate record. Trading lost data for occasional duplicates is the right tradeoff, and in any case, those duplicates would be dropped if Kafka’s idempotency feature is enabled.

Idempotent Producers

Supporting idempotent producer migrations raised two further problems. The first is that a producer’s identity has to be valid on both clusters. An idempotent producer’s ID is allocated by the cluster it connects to, but during a migration its writes must be accepted by the source while the topic is proxied and by WarpStream after the cutover. An ID issued by one cluster cannot simply be reused on the other, where it may collide with an ID already allocated to a different producer. So when a producer initializes, the Agent obtains an ID from the source cluster as well as from WarpStream and then stores the pair as a mapping in the WarpStream control plane. The producer only ever sees its WarpStream identity, and the proxy swaps in the source ID as it forwards each batch, so each cluster only ever sees the producer identity it issued itself.

The second problem with supporting Kafka’s idempotent producer feature is the sequence state that idempotency depends on. To enforce it, a Kafka broker tracks the sequence number of the last batch it accepted for each tuple of {producer_id, producer_epoch, partition} and rejects anything out of order. Until the cutover, that state lives on the source cluster, and WarpStream has none of it, so if WarpStream began enforcing idempotency from a blank slate, every migrated producer would arrive mid-sequence and be rejected with an OutOfOrderSequenceException.

For the migration to be seamless, WarpStream’s sequencer has to be hydrated with where every producer left off, just as Orbit already syncs consumer group offsets to carry over where every consumer left off. Every Kafka client tracks its own sequence numbers and carries them across the cutover unchanged, so when a producer’s first batch for a partition arrives after the cutover, WarpStream adopts its sequence number as that producer’s starting point and enforces the contract from there.

The blocking described earlier is what makes this safe. Because no write is acknowledged once a topic is cutting over, a producer’s sequence counter cannot advance during the cutover, and the number it presents afterwards picks up exactly where the last acknowledged write left off.

The one twist is ordering. An idempotent producer can have up to five batches in flight at the moment of the cutover, and in rare cases they can arrive out of order, leaving the sequencer seeded from a batch that was not the producer’s oldest. When that happens, the client eventually bumps its producer epoch and resends everything it has not had acknowledged. The cost of that recovery is at worst a duplicate record, but never a lost one, since a client never discards an unacknowledged batch. In our continuous testing, which runs full migrations end to end around the clock, the common client libraries pass through the cutover without surfacing a single error to the application.

Getting Started

Orbit Auto Migration is available today, on Agent versions v826 and above. If you already run Orbit, there is nothing new to install. Just enable Auto Migration in your Orbit configuration, pick a topic, and take it through a migration end to end. The full details are in the docs here. You can learn more about Orbit Auto Migration in this video:

Eternal Ltd. is an India-based technology company building a diversified ecosystem across food delivery, quick commerce, supply chain, and lifestyle services through Zomato, Blinkit, Hyperpure, and District, respectively. Its businesses span food delivery and dining, quick commerce, restaurant supply, and going-out experiences, serving millions of customers, merchants, and partners through technology-driven platforms.

Powered by technology and data, Eternal builds scalable products that enhance customer experiences, optimize operations, and enable smarter business decisions. As it continues to grow, Eternal remains focused on innovation, operational excellence, and sustainable long-term growth.

‍Cost Creep on Self-Hosted Apache Kafka® and Switching to WarpStream

Eternal used self-hosted Apache Kafka® as its main logging platform to store its logs and then feed them to ClickHouse for logs management.

To avoid networking fees that can quickly stack up due to crossing interzone boundaries and from multiple levels of replication, Eternal kept their self-hosted Apache Kafka setup constrained to a single availability zone with one level of replication.

Knowing that costs were higher than they needed to be and that the use case (logging) was not latency sensitive, Eternal completed a proof of concept of WarpStream’s Bring Your Own Cloud (BYOC) to replace this self-hosted Apache Kafka setup with WarpStream.

Eternal created a POC WarpStream cluster and dual-wrote the production logging traffic to their existing Apache Kafka setup and WarpStream. The entire proof of concept only took two weeks and another two weeks to move to production.

The result? A 60% cost savings vs. self-hosted Apache Kafka running in a single availability zone and they no longer have to worry about replication levels (WarpStream does not need to replicate), interzone networking, or cloud disks by leveraging S3-compatible object storage.

Eternal’s workload is very temporal — it drops to 2.3 GiB/s throughput at the lowest point of the day (when people are sleeping and not ordering) — and spikes to 25 GiB/s throughput (across multiple clusters) at its peak. WarpStream allows them to auto-scale their cluster throughout the day and make sure it’s always perfectly right-sized — something that was impossible on their prior, self-hosted Apache Kafka setup.

“By migrating from self-hosted Kafka to WarpStream for observability and logging, we’ve drastically reduced costs while maintaining reliability, thanks to quick auto-scaling and S3 storage with zero inter-AZ charges,” said Rajat Taya, a Senior Engineering Manager with Eternal.

Further Cost Savings by Replacing Managed Kafka

Eternal did not stop there as its Kafka architecture stretches beyond self-hosted Apache Kafka, and they wanted to see where else implementing WarpStream would produce benefits.

Managed Kafka was used for benchmarking — this is where Eternal’s backend services record live traffic requests into Kafka and then replay them later to measure performance and perform load testing.

Aside from costs due to cloud disks and interzone networking, a major issue with managed Kafka was cluster creation time; it can take as long as 20 to 30 minutes. Eternal replaced this managed Kafka cluster with WarpStream, which reduced cluster creation time to seconds and also enabled them to take advantage of WarpStream’s “scale to zero” functionality for periods where the cluster is idle.

“This is where WarpStream shines,” noted Taya.

Not only did Eternal simplify operations and eliminate these cluster-creation wait times, they achieved an 80% cost savings by replacing managed Kafka with WarpStream and were able to port their benchmarking — which averages 52 million messages per minute during a backfill — to WarpStream.

Architecture and Operational Flexibility

WarpStream was a perfect fit for Eternal’s latency-insensitive workloads referenced above, i.e., logging and analytics, and benchmarking. Eternal has some workloads that are more latency sensitive and for that, they leverage Confluent Cloud.

As you can see in the architecture diagram below, it’s not an either-or decision that Eternal had to make. They can treat Kafka topics like a service, deciding when data should flow to WarpStream or Confluent Cloud based on their cost and latency requirements. This enables them to achieve maximum cost savings and simplify operations while meeting their varying performance requirements on a workload-by-workload basis.

“Switching to WarpStream was a game changer. It made handling massive data streams effortless and freed our team from constant infrastructure headaches. Now, they can focus on more meaningful work. Plus, WarpStream’s smooth Kafka integration made the whole transition painless and future-proofed our observability setup,” said Nishant Sarraff, a Software Development Engineer at Eternal.

Make It Work, Make It Correct, Then Make It Fast

When WarpStream was first announced, we framed it as a significantly cheaper and easier to use implementation of the Apache Kafka® protocol if your workload could tolerate higher latency. The product was well received, but that was a very big if!

Since then, we’ve iterated relentlessly to improve the latency profile of the product to make it suitable for a wider variety of use cases. We made the Agents faster, reduced the latency of our control plane, built Lightning Topics, and added support for S3 Express One Zone. With these improvements, a WarpStream cluster tuned for low latency can achieve a P99 Produce latency of less than 50ms with a P50 of less than 35ms.

These latencies represent an order of magnitude improvement in latency since we launched V1 of the product and make WarpStream suitable for 99% of the world’s Kafka workloads without compromising on cost or ease of operations.

However, there was still one big gap in our low-latency story: GCP. In order for WarpStream to achieve a low-latency profile, it needs a low-latency storage medium. In AWS, this became available with the launch of S3 Express One Zone in November of 2023. In Azure, the “premium” (low-latency) tier of blob storage has been available since March of 2019.

GCP Rapid Buckets

But what about GCP? Well, for a long time there wasn’t a low-latency object storage product available in GCP, so we had no way of offering a low-latency GCP version of WarpStream. That changed in May of 2026 when GCP GA’d their new “Rapid” Buckets. GCP Rapid Buckets expose a similar API as a “traditional” regional GCS bucket, but they have a few differences also.

First, Rapid Buckets offer much lower latency than GCP traditional buckets, hence the name. This was exactly what we needed.

Second, Rapid Buckets only store data in a single availability zone. While that may sound like a show-stopper for a product like WarpStream that prides itself on delivering 11 9s of durability, in practice it’s a non-issue because WarpStream already supports storing data in a quorum of zonal buckets (a feature we developed specifically for S3 Express One Zone when it was first launched).

Third, Rapid Buckets charge significantly more money for every GiB of data stored. This is especially problematic because to achieve high durability, systems like WarpStream need to store the data in at least two different Rapid Buckets as discussed in the previous paragraph. This makes the effective storage cost of Rapid Buckets more than 10x higher than a traditional regional GCS bucket.

Again, this sounds like it would be a deal-breaker for a product like WarpStream that prides itself on minimizing cloud infrastructure costs, but in practice it’s also a non-issue because WarpStream is a distributed log-structured merge tree (LSM) that performs background compactions. As a result, WarpStream can be configured to land newly ingested data in a quorum of GCP Rapid Buckets to minimize latency, but then asynchronously compact those files into a regional GCS bucket a few minutes later. As a result of this architecture, our GCP customers get the low-latency benefit of Rapid Buckets without the correspondingly high storage costs.

There was just one problem: in our testing, GCP Rapid Buckets didn’t actually seem any faster than traditional GCS buckets. Specifically, when we benchmarked the latency of creating a new file, writing a few hundred KiBs of data to it, and then closing the file, we found that Rapid Buckets and traditional buckets had about the same latency of 80–100ms.

When we investigated more closely, it became obvious that the intended usage of Rapid Buckets was to use long-lived files that were regularly appended. Once a file was created / opened, appending to it was very fast, on the order of 1ms, but it appeared that the latency of creating a new file and “finalizing” (marking as sealed / immutable) an existing file had not been optimized at all. This was quite different from S3 Express One Zone, where creating a new file, writing some data to it, and then closing it was extremely fast.

There was an obvious solution to this problem: we could rearchitect WarpStream to have a mode where it used long-lived files and appended to them instead of constantly creating and flushing new files, but this would have been a large, invasive change that we hadn’t budgeted for, so we shelved the project.

A Few Weird Tricks

A few months went by and the request for low-latency WarpStream clusters kept piling up. We decided to talk to some engineers at GCP to see if there was anything they or we could do that wouldn’t require rewriting half of WarpStream. When we met with the engineers at GCP, they confirmed our suspicion that the reason we weren’t seeing any meaningful latency improvements was because our workload had two critical-path metadata operations: (1) opening the file and (2) closing the file, and those operations weren’t any faster with Rapid Buckets than they were with GCS buckets.

There was disappointment on both sides. We really wanted to take advantage of Rapid Buckets’ low-latency capabilities without a big rewrite, and the GCP engineers clearly felt like this was an important use case they wanted to address.

“You could just not close the files,” one GCP engineer mused.

“Wait what?” I said, sitting up straighter. “Can we do that?”

— “Yeah, definitely. Unfinalized files are readable and you can always read your writes. You’d just need to make sure that all your writes had been flushed successfully, but flushes are super fast, on the order of 1ms of latency.”

— “OK, so an unfinalized file will just look like any other file? It will be visible to GET requests and LIST requests? It won’t show up as a failed multi-part upload or something weird like that?”

— “Correct, the only difference between a finalized and unfinalized file is that a finalized file is immutable and won’t accept further appends if you’re using our gRPC interface. If you’re using the JSON/XML interfaces and don’t close the file, then it may take some time for the CRC/Size of the file to be updated.”

— “OK, that’s perfect then. I’ll have to make sure we never try to append to a file that we’re done with, but our system already never does that, I can just add a few extra safeguards to be sure. So, if I do this, how fast will it get?”

— “Well, that will eliminate one of the two critical-path metadata operations and since the latency of metadata operations dominates your workload, it should be about half the latency of regular GCS.”

— “OK, that’s not bad, but it would be really nice to get rid of the first metadata operation also.”

We both stared at each other, clearly formulating the same idea at the same time. “What if I just maintained a pool of pre-warmed files, would that work?” I asked tentatively.

“Yes, that would one hundred percent work,” the GCP engineer replied immediately and enthusiastically.

We had a plan, and it might just work! We agreed that I would go test our hypothesis and report back. To confirm the performance characteristics worked the way we expected, I wrote a short test program that looked roughly like the following:

client, _ := storage.NewGRPCClient(ctx, experimental.WithZonalBucketAPIs())
bucket := client.Bucket(bucketName)
payload := []byte("0123456789")

// Phase 1: open N writers (create-if-not-exists), flush to establish each handle
var writers []*storage.Writer
for i := 0; i < fileCount; i++ {
 w := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).
  If(storage.Conditions{DoesNotExist: true}).
  NewWriter(ctx)
 w.FinalizeOnClose = false
 writers = append(writers, w)
}

// Phase 2: write + close each open writer
for _, w := range writers {
 start := time.Now()
 w.Write(payload)
 w.Close() // Doesn't finalize file because w.FinalizeOnClose = false
 fmt.Println("write+close:", time.Since(start))
}

// Phase 3: read them back
for i := 0; i < fileCount; i++ {
 r, _ := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).NewReader(ctx)
 contents, _ := io.ReadAll(r)
 r.Close()
 fmt.Printf("%q\n", contents)
}  

Unfortunately, when I ran this program the reported latency for write + close was on the order of 80–100ms, and we really needed it to be sub-10ms for this to work. A few minutes later I realized that I’d forgotten one key detail that the GCP engineers had told me: the GCP clients are lazy and so my Phase 1 for-loop was not actually pre-warming the files, it was just creating an array of in-memory objects and my program would pay the full latency penalty of opening the file on the first call to Write().

I updated my program with one additional line, changing the pre-warming code from:

w := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).
    If(storage.Conditions{DoesNotExist: true}).
    NewWriter(ctx)
w.FinalizeOnClose = false
writers = append(writers, w)

And voila, the reported latency was now about 9ms, exactly where I needed it to be. I reported this back to the GCP engineer and he notified me that while reproducing my workload, they’d found a few inefficiencies with how the first write to a new file was handled and were making some improvements on the backend. When they notified me they’d rolled out the improvements, I ran my test program again and this time it reported latency of about 6–7ms. Nice!

Finally, I sent my entire test program over to the GCP engineer to confirm it was correct and he pointed out that it would be faster if I replaced my terminal w.Close() with a w.Flush() because Close() performs some extra work that is not required for my workload like accelerating how fast the size/CRC updates will be made visible to JSON/XML clients, as well as performing some client-side and server-side cleanup. That said, it was still important that the process called Close() eventually to ensure client-side resources were cleaned up, which was simple to implement with an asynchronous goroutine.

In the end, the final program ended up looking like this:

client, _ := storage.NewGRPCClient(ctx, experimental.WithZonalBucketAPIs())
bucket := client.Bucket(bucketName)
payload := []byte("0123456789")

// Phase 1: open N writers (create-if-not-exists), flush to establish each handle
var writers []*storage.Writer
for i := 0; i < fileCount; i++ {
    w := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).
        If(storage.Conditions{DoesNotExist: true}).
        NewWriter(ctx)
    w.FinalizeOnClose = false
    w.Flush()
    writers = append(writers, w)
}

// Phase 2: write + close each open writer
for _, w := range writers {
    start := time.Now()
    w.Write(payload)
    w.Flush()
    go func() {
        // Doesn't matter if this fails, just cleaning up some client
        // resources.
        w.Close()
    }()
    fmt.Println("write+close:", time.Since(start))
}

// Phase 3: read them back
for i := 0; i < fileCount; i++ {
    r, _ := bucket.Object(fmt.Sprintf("%s_%d", keyPrefix, i)).NewReader(ctx)
    contents, _ := io.ReadAll(r)
    r.Close()
    fmt.Printf("%q\n", contents)
}

With all of these tweaks and improvements the program was now reporting a latency of 2–3ms. Almost 40x better than the 80ms we started with! So I set about translating my test program into concrete changes in WarpStream to take advantage of these tips and tricks.

Summarizing the tricks (I know I lied in the title when I said it was just one):

1. Don’t call NewWriter() just-in-time when we’re ready to flush a new file. Instead, maintain a background pool of pre-created writers that can be retrieved on-demand.
2. The pool should also call Flush() on the writers before considering them “ready” to ensure that the file is actually created and the metadata operation completed so that the first real write to the object doesn’t incur that penalty.
3. Objects should not be finalized and we can ensure that by setting w.FinalizeOnClose = false.
4. “Close” the file by calling Flush(), not Close().

The last three tricks were trivial, a few lines each at most. But trick 1 was … tricky. Specifically, we had to juggle a number of different requirements:

1. The object pool always had to have spare files, otherwise a call to retrieve a file from the pool may block on creating a new file on the fly and cause a latency spike.
2. Files that had been open for too long should be thrown away / discarded. This is important for two reasons: one, to prevent a latency spike from trying to use a stale connection and two, also because WarpStream has a background reconciliation loop that scans the object store for files that aren’t tracked by the control plane and deletes them. To prevent this loop from deleting files that are “in-flight” we look at the creation time of the file and apply a grace window. Therefore, if a file had been in the pool for a period that was approaching the grace window, we couldn’t use it without risk of it being deleted in-between us writing it to the bucket and it being committed to the control plane (this would result in data loss).
3. The pool should refill itself at a rate roughly equal to the rate at which it was drained.
4. The pool should not create and throw away an excessive number of files as this would be expensive from both a performance and cost perspective.

Luckily we already had an abstraction in the codebase called a LoadingQueue that was almost exactly what we needed:

// LoadingQueue is a FIFO queue abstraction that abstracts away
// the process of fetching items to load into the queue.
// 
// It allows the user to provide a load function and then just
// call Get() in a loop and the queue will automatically handle
// refilling itself in the background.
type LoadingQueue[T any] interface {
   Get(ctx context.Context) (T, error)
}

The implementation of the LoadingQueue itself is quite simple. You can think of it as a light abstraction over a buffered channel in Golang where a Goroutine calls a user-provided LoadFunction in a loop and inserts the result into the channel. When the channel is full, backpressure is applied naturally to the goroutine running the LoadFunction.

We made just three simple changes to the configuration of the LoadingQueue struct to support our new use-case:

1. We added an optional DiscardFunction parameter which is a user-supplied function that is called on any object that ends up being discarded (I.E not returned in a call to Get()). In the case of the Rapid Buckets implementation, this function just calls Close() on the file.
2. We added a MaxItemAge parameter that limits the age of an object that will be returned by the loading queue. Any object older than this parameter is automatically discarded using the DiscardFunction.
3. We added a LoadConcurrency parameter that controls the maximum number of Goroutines that will run the LoadFunctionconcurrently. This was necessary because the latency of pre-warming files was on the order of 80ms, which meant that we needed more than one Goroutine doing it continuously to keep Agent VMs with many cores satisfied.

The Final Result

Once that was done, we could finally measure the actual impact on latency in one of our test clusters:

With these changes in place, the P99 latency to flush a 1MiB file with Rapid storage was about 33ms and the P50 was 20ms, about 2–3x faster than with a traditional GCS bucket. The end result didn’t end up being 40x faster like in the previous example. This is because the gap between Rapid and traditional GCS buckets drops when the amount of data written increases, and also because this measurement includes a few non-bucket related operations like compression, which are performed streaming and whose latency can’t be disentangled from the latency of writing the file. Still, a 3x reduction in P99 is nothing to sneeze at!

‍Note: The Rapid Buckets have an important limitation: they are not compatible with WarpStream’s Lightning Topics feature. This is a temporary limitation due to uninteresting details of how lightning topics are implemented. This limitation will be fixed in the next few months.

If you want to get started with WarpStream and GCP Rapid Buckets, check out our documentation.

What’s the best compliment you can give a piece of infrastructure? According to Maor Bril, who leads data infrastructure and several Labs products at Character.AI, it’s that the product is boring.

Character.AI is one of the largest AI entertainment platforms in the world, with roughly 60 million monthly active users. To keep that experience safe and to constantly experiment with new prompts and models, the team needed to capture not just user messages, but the entire prompt sent to the LLM (system templates, character definitions, memory summaries, etc.) in near real time, at enormous scale.

They knew they wanted the Kafka API (their stack already spoke Spark, Trino, and ClickHouse). What they didn’t want was the cost and operational pain of running Apache Kafka® in the cloud the old way. That search led them to WarpStream, and the rest of this episode of Confluent’s Life Is But A Stream tells the story.

Interested in the full, technical case study of how Character.AI uses WarpStream? You can read it here.

“It Just Works”

WarpStream is the most boring product I’ve ever got to work with. In infra, that’s the biggest compliment you can ever give someone. It just works.

That theme echoes what we heard from Cursor in an earlier episode (“Data goes in, data goes out. It just works”). When you’re a lean team, boring infrastructure is exactly what you want — it frees you to focus on the things that actually differentiate your product.

Kafka, Without the Kafka Pain

Maor had managed big Kafka clusters earlier in his career, so he knew the tradeoffs going in. WarpStream gave his team the Kafka ecosystem they already relied on, minus the parts nobody enjoys, like rebalancing partitions, adding brokers, and babysitting stateful clusters:

It sounded too good to be true. It looks like Kafka, it talks like Kafka, but it doesn’t have all the pain points of Kafka.

Because WarpStream Agents are stateless and run on Kubernetes behind an HPA, they simply scale up and down with the workload. And because everything is backed by S3-compatible object storage, durability and multi-region replication come for free.

~90% Cheaper, With No Extra Headcount

When the team scoped out the numbers against the usual players, the gap wasn’t subtle:

It’s a fraction of the cost of anyone else — about 10 to 15% total cost for a year. [That’s an 85 to 90% cost reduction.]

Normally, that kind of Kafka cost reduction usually means more DIY toil and more engineers to manage it. With WarpStream, Character.AI got the savings without giving up features or adding operational overhead.

A big reason is where the Agents run. As an AI lab leasing H100 nodes, Character.AI already pays for far more CPU than inference needs:

Those CPUs are effectively free CPU compute that WarpStream Agents are running on. … For us it was a win-win-win: cheap S3-compatible storage, our own BYOC compute we’re already paying for, and all the data stays in our own project.

No third-party vendor in the data path, no new infrastructure to introduce, and in-cluster traffic means no surprise egress bills — even when you’re moving a lot of data.

BYOC That Legal and Security Actually Like

Because WarpStream is BYOC, the Agents run inside Character.AI’s own VPC, and data governance got dramatically simpler:

No user data ever leaves our own domain. That makes the whole story very easy to sell to our legal and security team. It was a win-win-win across the board.

5 GiB/s, and It Just Works

The headline number isn’t just cost. With WarpStream’s Managed Data Pipelines (Bento running directly inside the WarpStream Agent), Character.AI built their processing pipelines in the same system that moves the data, and pushed it hard:

We partnered with the WarpStream team on some custom connectors and built a bigger pipeline serving 5 gigs per second. And it just works.

That single pipeline now powers real-time trust and safety classifiers, prompt/model A/B testing, replay-on-failure debugging, trend analysis, and even ETL and reverse-ETL into a feature store (via a Bento connector the team contributed back to Bento).

Try the Thing

One last reason WarpStream made it past the “too good to be true” filter: it’s trivially easy to evaluate.

With WarpStream, you can run it on your laptop in one command line. Functionally it works — that’s awesome. Let’s run this.

If you could prove out a streaming platform in a few minutes instead of a multi-week POC, why wouldn’t you?

WarpStream is boring. And, as Character.AI will happily tell you, that’s the highest compliment infrastructure can earn.

I am a product person. I have always struggled to describe my job because the answer is literally “I don’t know, it depends.” I am not a software engineer, and while I’ve dabbled in the dark arts from time to time, I have never been paid to write code.

In the last few months that’s changed.

Most recently, I implemented a new Chargebacks feature that exposes approximately how much of an invoice’s usage is attributable to a given Kafka topic, Tableflow table, or Schema Registry subject. I was going to spend a bunch of time explaining in detail how Chargeback calculations work, but if you want to know more about that, just go read the docs. We’ve got bigger fish to fry today.

How I Built This Is More Interesting Than What I Built

I built this feature myself (with a lot of help from your friendly neighborhood frontier models and Cursor). The code touched some pretty sensitive parts of the codebase. I had to be careful to not mess things up. I had to learn a lot about things that were unfamiliar to me. I had some ideas that were, in retrospect, not good. I asked questions (of people) and had multiple models and multiple people check my work. I explained some key details about how it works to other people. Then I deployed it incrementally in staging and tested it. Then I gave a couple of customers access, and asked them to test it. They found a couple of bugs. (Thanks Aly from PostHog!)

Finally, this morning, I deployed the feature flag that turned it on for everyone, otherwise known as “going GA.”

Much has been written about how AI is eating the world and rendering humans obsolete. All I can say is that things are possible now that were not possible six months ago. Does that mean that software engineering is obsolete and everyone is a PM because now it’s all about what gets built, not writing code by hand? Or does it mean that PMs are no longer needed because engineers can get all the “product sense” that they need from an infinite well of information served up by the latest model?

I think it’s a lot more nuanced than that. The old ways of defining roles are dead. Now, everyone exists on a spectrum. Maybe there are some broad categories that still make sense to divide people on (I am much more qualified to be pretend-software engineer than a pretend-lawyer, and actual meat-space jobs like surgeon and astronaut are a no-go). But I’m skeptical of the idea that “product management” and “software engineering” are actually different roles at all. Individual people can exist on one extreme end of the PM/SWE spectrum or the other, but they’re on the same spectrum. Maybe they always were.

One thing is very obvious to me: some teams are much more successful at getting better and faster outcomes using LLM-generated code than others. I think I’m starting to understand why.

Systems Matter at Least as Much as Tools

A hammer is a tool that’s designed to be pretty good at driving nails into wood. Anyone can whack a nail into a board with a hammer, but if you’ve ever watched a professional builder whack nails into boards, you know that there’s a vast difference between an expert hammer user and a novice. LLMs are like air-powered nailguns. The act of swinging a hammer to drive nails has been automated. You can whack a lot more nails into a lot more boards much faster using a nailgun than a hammer, but that does not make you any better at actually building houses.

The tools have changed, and this change has enabled many of us to contribute an unprecedented amount of output. But output was never the goal. You are not a better builder of products just because you can now feed your PRD to an Agent and dump a pile of vibeslop on the heads of unsuspecting code reviewers a few minutes later. Subject matter expertise still matters. Taste matters. Understanding what you are contributing, how it works, what tradeoffs you have made, which alternatives you have rejected, is harder now than it’s ever been. But it’s never been more necessary.

On the WarpStream team, one way that we ensure quality and craftsmanship is by requiring that every pull request adheres to a set of “affirmations.” You have to check the boxes before you can get a review. It feels bad if you ask someone to review something after falsely affirming to yourself that you understand what you are asking them to review. This accountability keeps us honest. If I can’t check these boxes with confidence, I feel shame, and the dopamine hit I get from submitting a +2343 line PR won’t make up for it.

In addition to the PR affirmations, we also have a document defining guiding principles and expectations for our SDLC that we all agree to adhere to. An excerpt:

“We don’t care how you generate code. Write it by hand, or spawn 20 sub-agents to do it in parallel. This is your own business. However, we do expect that you’re willing to stand behind the code that you put forward. This means that at the end of your process, whatever that process may be, the generated code should be of similar or higher quality to what you would have generated had you written it all by hand.”

We also have an extremely robust CI pipeline, and comprehensive test coverage. We have some rules that are codified in CI checks as well, such as “you cannot define a feature flag and also set that feature flag in the same PR” and “you cannot make both protobuf definition changes and other changes in the same PR”. Rules like these ensure that we don’t make changes that are known to be dangerous and difficult to roll back.

Our CI pipeline kicks off an AI code review on every push, so before a PR gets to a human reviewer, it’s been reviewed at least once, by a review bot that’s been given pretty specific instructions on how to make sure we don’t merge garbage. It usually does a pretty good job of finding obvious bugs and performance improvements. I also like to get an AI code review from multiple models before I even put my branch up for the CI bot to run against. (This is so easy to do with Cursor that I’m actually pretty convinced that none of the frontier model providers have any pricing power at all, which brings up many terrifying-but-academically-interesting questions about the economic viability of offering models as a service — but I digress.)

If you can’t tell by now, our team is extremely AI-pilled. I am extremely AI-pilled. These things are really useful. But they are still just tools.

What Is Old Is New Again

Old truths about what makes an effective product builder remain the same, regardless of the tools that are available to us, or the titles that we give ourselves.

Agency is, in my opinion, the single most important defining characteristic of “people who are good at building stuff.” I don’t think this is controversial. Agency means that you have a drive to do things. Others have referred to this as “bias for action” or “getting shit done”, but I think it’s actually more than that because those phrases and taglines emphasize output. Agency isn’t about output. People with agency are motivated by affecting outcomes, not just producing more widgets at the widget factory.

Another defining trait of the Product Builder is their ability to exercise good judgment. Good judgment comes from the ability to think critically about a problem, and come to a conclusion independently based on the available information. People who have good judgment typically have strong convictions because they’ve thought somewhat deeply about a problem from first principles, and therefore have a point of view. Sometimes their points of view are uncomfortably contrarian. Sometimes they perfectly uphold the consensus. If you’ve thought critically about a problem, and you’re honest with yourself about what you do and don’t know, then you can confidently articulate what you think. Good judgment requires critical thinking.

Good judgment also requires that you understand your users and customers. You have to know who you’re building for, why they care, and what matters to them. On the WarpStream team, we put all of our engineers on customer calls all the time. This practice has built a huge amount of product sense on the team. It has improved every single thing about our product, and our GTM motion. You should do this, too.

Agency and exercising good judgment have always been assets for product-focused people. But these days, I think they’re more relevant than ever before. The barrier to entry is gone, and as a result it’s easier than ever to ship mediocrity. Once you open the door to mediocrity, it cannot be closed. To avoid this descent into despair and doom, you must create systems that fight entropy. You must also build a culture that emphasizes quality and conscientiousness over quantity and output. Again, this is not new. It’s just more important now than ever before.

You Are Not the Sum of the Tools That You Use

Coding agents are pretty cool. This Chargebacks feature would not exist if I didn’t have Opus 4.8 and GPT 5.5 working for me. I wouldn’t have prioritized it. It simply would not have made the cut line for a “real” engineer to work on, and I hadn’t learned enough Go to write it myself in a reasonable amount of time.

But am I offering yet another breathless take about how software engineering is dead and human engineers won’t exist a year from now? No.

If I didn’t have humans to review my code, or teach me about dependency injection, or tell me that some of my proposed changes “seem insane” (sorry, no tweet link for that one), I might have shipped something, but it probably would have been bad. It could have caused major problems, up to and including bricking customers’ WarpStream clusters. Skill issue? YES. That’s the whole point.

Nearly all of our engineers spend substantial time working with coding agents. I don’t know the actual percent of our codebase that’s been AI-generated, but it’s large. If we didn’t have an extremely robust culture and set of tools specifically aimed at controlling the entropy increase that LLMs introduce, our product and codebase would likely be regressing into mediocrity despite everyone’s best intentions. And I can tell you first-hand, that’s not what is happening.

Your Reflexes Are Wrong

It’s tempting to think that once the models get good enough, we can simply replace all the human engineers with LLMs, just like it’s tempting to think that “Product Led Growth” means replacing all your salespeople with a signup page. This is the wrong reflex. It’s not even that “the models aren’t ready.” They’re never going to be “ready”, as long as they’re LLMs. Scientifically speaking, I don’t know what gives your brain the ability to express good taste and exercise good judgment, but I’m pretty sure that predicting which character is most likely to come next in a sentence ain’t it. More people can do more with LLMs, but LLMs can’t do everything.

I’ll leave you with a hot take about the advent of LLMs and where I think this is going spectacularly wrong right now. People with agency and good judgment do not need to be told which tools to use. If you have to put up a token leaderboard and tell everyone at your company at an all-hands meeting that they’re being left behind if they don’t use the LLMs to make more widgets, your problems are way, way deeper than whatever tools your people are using.

Forcing people to use a specific tool or category of tool only exposes deficiencies in your culture, and by extension, your leadership. Think twice before you hand out nailguns to your hammer-wielding framers and tell them that they’re going to be judged based on how many nails they shoot into boards. You won’t get more or better houses that way. The output is not the point. Counting tokens, or even counting adopters, is completely the wrong thing to do. Stop doing it.

So a PM Vibecoded a Feature, Big Deal

Maybe none of this matters to you, and reading this was a big waste of time. Hopefully, though, we can all start to agree that the collective AI psychosis gripping our industry has gone a bit too far, and we need to reset our expectations about what these things are actually good for.

A lot of people are going to read this and think, “Wow, they’re letting a PM contribute code to production? Good luck with that.” And that’s probably a pretty good default position to take. If you don’t have all the systems, processes, and culture already set up to support people messing around in production, you will be in for a wild ride. But we do have those systems, processes, and culture, which is what enables me to do this at all. Getting hands-on with our codebase has made me better at my craft, just like putting our engineers on sales calls has improved their ability to build meaningfully valuable features and functionality for our customers.

But I am not a replacement for any engineering headcount, and I probably never will be. I’m just able to do more without needing someone else’s help. Just like a builder who swaps out his hammer for a nailgun.

Almost two years ago, we published a blog post about how we expanded WarpStream from a single region to multiple, while still presenting it as a single pane of glass. Since then we’ve added a lot more regions.

The reason for this is simple: WarpStream customers want control planes as close as possible to their workloads, ideally in the same cloud region. As a result, we have WarpStream control planes in 24 unique regions across three different cloud providers.

Why do customers care which region their WarpStream control plane is running in? The obvious answer would be latency, but that’s only a small part of the story. The much more important reason is about reliability and alignment of failure domains. For example, someone running all their services in AWS us-east-1 shouldn’t have to worry about how their workload will be impacted if GCP has an outage in any region.

And last but not least, some customers want to be completely isolated from others, and are asking for what we call “dedicated control planes”. Under the hood, it means that the customer is using a completely separate infrastructure — from the load balancer to the control plane containers.

All of this is to say that since we wrote that first blogpost we went from running WarpStream’s control plane in 4 AWS regions to 24 regions across 3 different cloud providers. Like the Agents, WarpStream’s control plane is completely stateless and offloads a lot of the trickier operational problems to cloud provider primitives like DynamoDB and Spanner, so operating this many control plane regions is really not that much of a burden for our team. The control plane mostly just works and we rarely get paged or have to perform manual operations.

While operating this many control planes is pretty manageable, deploying this many control planes is a completely different story. In this blogpost, we are going to explain how we managed to get there without introducing too much friction for our engineering team, without deploying any overly-complicated technology, and without hiring a dedicated dev-ops or infrastructure team (despite my lobbying for it!). We will also take the opportunity to explain our infrastructure layout, and how we try to deploy slowly and safely any change.

WarpStream’s Control Plane Architecture

Before we get into the specifics of multi-regions, let’s take a step back and explain the high level architecture of our control plane regions. Generally speaking, we refer to a deployment of WarpStream’s control plane as an “environment” (an environment is the same thing as a “cell” if you’re familiar with that terminology instead).

A given cloud provider region can have one or more WarpStream environments:

Each environment is completely isolated from the others and they have no awareness of each other or ability to communicate with each other. Each environment is fully isolated all the way down from the ingress load balancers to the backing storage, as well as everything in-between.

In each environment, we run three separate “zonal” Kubernetes clusters, one per availability zone.

Let’s pause here for a second because that’s pretty different from what most people do. We decided to deploy three different “zonal” Kubernetes clusters, with all resources in that Kubernetes cluster located in a single availability zone, instead of deploying a single multi-AZ regional cluster. We chose this model for a few reasons:

1. It aligns our failure domains with the cloud provider’s so it’s easy to completely shut down a zone if there is a zonal outage.
2. It enables the environment to survive a “fat finger” problem where any entire Kubernetes cluster is broken or destroyed.
3. It makes all types of Kubernetes operations (like upgrades) significantly less scary and burdensome. In the absolute worst case scenario we can blow the entire Kubernetes cluster away and rebuild it from scratch with no customer impact.

The last missing piece we need to explain is that in addition to our primary control plane service, each environment also runs a “control plane shadow” service. The shadow service is used to test new versions of the control plane service by dual-processing a percentage of all production traffic on the new version and making sure it generates the exact same output as the previous version.

A simplified version of the flow looks like this:

We could write an entire blog post about the shadowing service so we won’t go into further detail here, but one point we’d like to belabor is that the shadow service is not a canary. Customer Agents always receive a response generated by the actual control plane service. This is important, because it means we can detect regressions in production without impacting any customers.

When we have a new control plane image to rollout, we always deploy the shadow service first. Any discrepancy between the shadow service and the real control plane service responses triggers an alert in our observability platform. Of course we have the ability to bake into the shadow service some “known discrepancies” when we roll a change that actually changes the response, so only unexpected differences page the engineering team.

Deployments

OK, now that we’ve laid the scene, we can finally talk about deployments. The first generation of our deployment tooling was just a Go program that ran on an engineer’s laptop and orchestrated a simple flow of:

1. Deploy to staging.
2. Wait.
3. Deploy to production.

This was suboptimal for a number of reasons, not least of which was that an engineer who started a deployment on a Friday afternoon but then allowed their laptop to go to sleep could inadvertently cause a production code rollback when they woke their laptop up on a Monday morning. Before you judge us too harshly, remember that this was back when the entire engineering department was 4 people (including the founders!) and the product didn’t even have a way to charge customers yet!

Obviously this wasn’t good enough, so before we GA’d the product we rewrote all of our deployment tooling with a number of important changes:

1. We migrated the tooling to run inside of GitHub Actions so it was no longer tied to any individual’s laptop.
2. We added a distributed lock using DynamoDB so that it was impossible to have concurrent deployments targeting the same environments.
3. We added a bunch of runtime safe-guards and checks to eliminate a whole class of common mistakes like deploying code to production that wasn’t merged to main, deploying code to production that was older than the code currently deployed, automatic pausing of deployments if monitors in any of the recently deployed environments fired, etc.

The “UX” of this new tool heavily relied on Slack: all deployments created a dedicated Slack thread, and the tooling posted regular updates as the deployment progressed. There was also a direct link to the GitHub Action if anyone wanted to cancel it. In order to be able to correlate deployments with other signals, our script would publish logs & metrics to our observability provider.

For example, this is what it looked like when we deployed an internal service to a staging environment:

Under the hood, a deployment consists of:

• Updating all the pods in one zonal Kubernetes cluster
• Waiting for a while and making sure no monitors trigger in our observability system
• Repeat in the next zonal Kubernetes cluster until we’re done

When you factor in the shadowing service, “deploying” the control plane to an environment consists of:

This new tooling was much better, safer, and served us well for a really long time.

Trains

Now that you are more familiar with how our services are deployed towards a single region, let’s take a peek at what it means when we started to expand WarpStream’s footprint all over the world.

When we started increasing the number of WarpStream regions, one thing became very clear: it wasn’t practical to require that every engineer slow roll every change they made through all of our production regions. We’d either have to make deployments run faster than we were comfortable with, or our engineers would only ever be able to deploy at most once per day, neither of which were acceptable.

As a result, we decided that we would use a deployment train strategy where engineers would merge their changes and then deploy them to a single region henceforth referred to as “the primary”. There is nothing special about the primary region other than the fact that it’s the region that receives new changes first (after they’re promoted through our staging and shadow environments first, of course).

New code would bake in the primary region overnight, and every morning an automated train would run that would query our observability system to determine which commit had been running in the primary region overnight, and then promote that version to all of our remaining production regions one at a time.

While this may sound complicated, the implementation was quite simple (a few hundred lines of Go). It was basically just a matter of taking our existing deployment tooling and throwing it into a for loop.

We scheduled a run of this train each business day at noon UTC inside a GitHub Action, and let it run. Just like regular deployments, it would create a thread in Slack, leaving messages at each step so we could monitor the progress.

Life was good — for a while: the engineering team was mostly unaffected by the new regions and the rate at which they were growing. Engineers would merge their changes and then deploy them to the primary region. And every day, we would run a train to “sync” the other regions. Of course, people still had to change their habits a bit:

• If we had an urgent bug fix to get out to a customer, then maybe we would need to take a shortcut depending on which region the customer was using. We could either use the deployment script to target a different region, or do a regular deployment and trigger manually a custom train that would target only this region.
• It was no longer possible to make the assumption that as soon as a single deployment was done all environments were up to date: non-primary regions always lagged a day or two behind.

Adding Layers to the Cake

Eventually as the number of regions grew to twenty and beyond, the train became a frequent source of flakiness. The longer a given train took, the more time there was for things to go wrong:

• The GitHub runner could crash.
• A spurious alert in one region could cause the entire train to be canceled even though the alert had nothing to do with the ongoing deployment.
• A transient failure in one of our deployment dependencies (GitHub, Slack, our observability provider, even the Kubernetes API) could cause the train to fail as well.
• A cloud provider temporary shortage of the required instance type in one availability zone.

We could restart the train, but all the state for the current train was stored in the processes memory which would be lost when it failed. We could “fast-forward” through completed steps on restart, and we even experimented with some small tricks like persisting the current processes state in DynamoDB on a regular basis and reloading it on restart.

This allowed us to make it easy to resume a train that stopped or failed for any reason:

But there were other issues. For example, GitHub Actions have a hard-coded maximum duration of six hours, but once we got past 20 regions, deploy trains took longer than that. That wasn’t a limitation of our deployment model either, we want production deployments to be slow so that we only ever update a single region at a time and our automated shadowing and monitor gates have time to catch issues between deployments to regions and individual availability zones.

We also ran into other pain points like the fact that implementing relatively simple features like: “pause deploy trains outside of engineering business hours” and “deploy this financial trading firm’s dedicated control plane outside of US trading hours” became unnecessarily difficult and painful with this approach.

To Trains v2 and Beyond

Our deployment tooling had grown from a 100-line Go program to a complex beast that was becoming more and more difficult to maintain. In addition, deployment train paper cuts were really starting to add up in terms of wasting engineering hours. Eventually, we decided that we’d accumulated enough technical debt and achieved sufficient scale that we needed a new solution.

At this point, we did consider leveraging open source or off-the-shelf tooling, but every tool we investigated seemed unnecessarily complex relative to our needs, and more importantly, we liked having complete control over our deployment tooling. As an infrastructure provider, our ability to roll out changes to dozens of unique environments and regions running complex and stateful software is extremely core to our business and we wanted to maintain full control of that process.

As a result, we settled on doing a full rewrite of the deployment tooling: trains v2! Starting from first principles, we decided to use a completely different architecture for our trains v2 implementation: async reconciliation. Reconciliation loops are one of our favorite distributed systems patterns, and we’ve discussed them extensively in other blog posts like the ones we wrote about Schema Linking and Object Storage Garbage Collections.

The high level design we ended up using is the following:

This is a classic reconciliation loop. The target state is defined by using a user who specifies an operation like: “deploy version X to service FOO in all production environments” and the CLI tooling converts that to an ordered list of target states and stores them in a DynamoDB table. A background lambda runs periodically, checks the status of each environment against what it’s supposed to be in the DynamoDB table, and kicks off GitHub runners to reconcile any differences.

This new architecture was not that different from what we previously had, so it didn’t take long to start submitting our first trains with this new framework.

With this new design, it was much easier to implement new features and quality-of-life improvements. For example, making it so deploy trains pause automatically outside of business hours was just a few lines of code in the new implementation. The reconciliation lambda just needs to check if the current time is within business hours at the beginning of each reconciliation loop, and if it’s not, do nothing.

Consider another quality-of-life improvement that was very difficult to support in trains v1, but trivial in v2: a deploy train is interrupted by a failing monitor gate. You investigate, assess the impact, and decide that it was either a flake or unrelated to your deployment so you want to move forward. The only thing you have to do is to respond to a Slack thread and the train system takes care of the rest. It will automatically put an appropriate downtime on the monitor, check that nothing else is triggered, and continue the deployment:

Another pain point in our setup was how we handled infrastructure changes. We have always been keen on keeping a human on the loop when it comes to provisioning critical infrastructure: Kubernetes clusters, load balancers, databases … everything needed to deploy our control plane all over the world. This meant that in practice that after going through the regular code change process — pull request/review/merge — the terraform apply step would be performed carefully by an engineer who would triple check the plan before applying. This worked fine when we only had a few regions, but as the number of regions grew it became quite painful to roll out even simple changes — like adding a tag — across all of our regions.

Luckily we took this into account when we were designing Trains v2, so it was very easy to reuse the existing framework to automate the process of rolling out a terraform change across dozens of regions. This also enabled us to apply more targeted changes (like changes only impacting our Kubernetes clusters) on a zone-by-zone basis and take advantage of all the other features built into the Trains functionality like automated monitor gating.

We even built some advanced “plan comparison” logic into the terraform trains functionality so that if the plan for a new region was identical to the plan for a previously-approved region (with the only difference between the two plans being the name of the region that the change was being applied to) the Terraform apply could proceed without an additional human review.

Our infrastructure isn’t static, so keep an eye on our blog for future updates about its evolution. You can subscribe to our newsletter to get blogs delivered to your inbox.

I typed one sentence into Cursor: “Is anything wrong with my WarpStream clusters?”

Fifteen seconds later, the AI had checked all 36 clusters and surfaced three issues I didn’t know about:

• A cluster silently approaching its partition limit
• An Agent running pipelines alongside Kafka operations
• A consumer group with a rebalance timeout short enough to trigger infinite rebalancing

That’s what WarpStream’s MCP Server makes possible. It’s a Model Context Protocol integration that connects AI assistants like Claude, Cursor, and Windsurf directly to your WarpStream clusters, so you can query cluster health, search event logs, inspect ACL decisions, and explore topics without leaving your IDE.

What Is MCP, and Why Does It Matter for Kafka Data Infrastructure?

The Model Context Protocol is an open standard that lets AI assistants call external tools and data sources on your behalf. Instead of copy-pasting log snippets into a chat window and hoping the AI can make sense of them, MCP lets the AI query your systems directly with the right context, in real time.

For a data streaming platform, this is a natural fit. The questions operators ask most often — “Is anything broken right now?”, “Who denied that ACL operation?”, “How many errors has my pipeline thrown in the last hour?” — are exactly the kind of structured queries that map cleanly onto a tool-based interface. The WarpStream MCP server makes those queries available that any MCP-compatible AI can call.

What You Can Do

The MCP server exposes a set of tools that cover the most common operational workflows:

These are just the highlights. The MCP Server exposes 30+ tools covering cluster configuration, consumer groups, ACLs, pipelines, and more. See the full list in the MCP Server documentation.

events_query is the most powerful of the bunch. It runs queries across Agent logs, ACL logs, pipeline logs, and data lake events using the WarpStream query language, and it’s where most real debugging questions get answered.

diagnostics_status is the second most powerful tool. It returns a snapshot of your cluster’s health — active issues with severity levels, failing checks, and cost optimization suggestions — the same data surfaces you’d find in the Diagnostics section of the Console, but now queryable by your AI assistant as part of answering a broader question. If you ask, “Is anything wrong with my cluster right now?”, this is the tool doing the heavy lifting.

list_topics is worth noting, too: it includes recently-deleted topics, which is useful when you’re investigating something that’s no longer there.

Everything else, from cluster discovery to Diagnostics to events configuration, is handled by the supporting tools the table above maps out.

A few examples of what you can now ask your AI assistant directly:

• “By how much are my consumer group falling behind?”
• “Did anything change in my cluster in the last 24 hours?”
• “Which principal has been hitting ACL denials this morning?”
• “Is my pipeline throwing errors, and if so, from which stage?”

The AI handles the query construction. You just ask the question.

Under the Hood

The MCP server is available at https://console.warpstream.com/mcp and uses standard HTTP transport, so it works with any MCP-compatible client. Authentication uses WarpStream API keys passed via HTTP headers, which are the same keys you already use to interact with the platform.

API keys generated from the MCP setup page are read-only and scoped to the specific workspaces you select, so there’s no risk of an AI assistant accidentally modifying anything. If you manage multiple workspaces, you can grant access to all of them using workspace-scoped headers in a single configuration, rather than maintaining separate setups per workspace.

Setup takes about two minutes. Navigate to the MCP section in the Console, select your workspace(s), and click Generate Configuration. The Console produces a ready-to-paste configuration snippet for whichever client you’re using.

For Cursor, that looks like:

{
  "mcpServers": {
    "warpstream": {
      "url": "https://console.warpstream.com/api/v1/mcp",
      "headers": {
        "warpstream-api-key": ""
      }
    },
    "warpstream-docs": {
      "url":"https://docs.warpstream.com/warpstream/~gitbook/mcp"
    }
  }
}

Claude Desktop, Windsurf, and Claude Code (via CLI) are supported out of the box as well.

Built on the Same Foundation as Events

If you’ve been using WarpStream’s Events feature, the MCP server will feel immediately familiar. The events_query tool uses the same pipe-separated query language:

from topic "" | time 1h | filter data.log_level == "error" | fields @kafka_ts, data.message | sort @kafka_ts desc | limit 50

The events_query tool requires Events to be enabled on your virtual cluster, which is a separate step from connecting the MCP server. If you’re already using Events for observability, you’re halfway there. If you’re not using Events yet, this is a good reason to take them for a spin.

Why This Matters

Streaming infrastructure is operationally complex. Agents, topics, ACLs, pipelines, and schema registry all have their own failure modes, and when something goes wrong, the last thing you want is to remember the exact query syntax for pulling ACL denial events from the last 30 minutes.

The MCP server doesn’t replace understanding your infrastructure, but it does reduce the friction between having a question and getting an answer, which is most of what debugging actually involves.

We think of this as a natural extension of the work we’ve done with Events and Diagnostics. It’s not a replacement for those features or tools, but a new way to access them that fits into how engineers are increasingly working.

That opening wasn’t hypothetical. The WarpStream team runs the MCP server against our own staging environments across AWS, GCP, and Azure. The partition limit issue above, the consumer group rebalance timeout, the embedded pipeline conflict — we caught them this way. This is how we debug now.

This is also not a static release. We’ll keep expanding the MCP server’s capabilities as we identify new opportunities, and as customers tell us what’s most useful to them.

Getting Started

The WarpStream MCP works with common dev environments: Cursor, Claude, VS Code, and others. Quick-start deep links are provided for Cursor and VS Code. For everything else, getting started is a quick copy-and-paste.

The MCP server is available now for all WarpStream accounts. You’ll need at least one virtual cluster and admin access to your workspace to generate an API key. Check out the MCP Server documentation to get started.

Making Life Even Easier for Our Customers

WarpStream’s BYOC architecture gives users the data governance benefits of self-hosting with the convenience of a fully managed service. Events are a new observability tool that take this convenience one big step further.

When we initially designed WarpStream, our intention was for customers to lean on their existing observability tools to introspect into WarpStream. Scrape metrics and logs from your WarpStream Agent and pipe them into your own observability stack. This approach is thorough and keeps your WarpStream telemetry consolidated with the rest. It works well, but over time we noticed two limitations.

1. Not all our users have modern observability tooling. Some may take structured logging with search and analytics for granted, while others don’t have centralized logs at all. For various reasons, some teams have to tail their Agents’ logs in the terminal.
2. High-cardinality data is expensive. Customers with modern observability systems often use third party platforms that charge a fortune for high-cardinality metrics. We restrict the Agent’s telemetry data to keep these costs low for our customers and in turn these restrictions reduce visibility into the Agent.

We realized that to truly make WarpStream the easiest to operate streaming platform on the planet, we needed a way to:

1. Emit high cardinality structured data.
2. Query the data efficiently and visualize results easily.
3. All while keeping the data inside the customer’s environment.

Events address all three of these issues by storing high-cardinality events data, i.e. logs, in WarpStream, as Kafka topics, and making these topics queryable via a lightweight query engine.

Events also solves an immediate problem facing two of our more recent products: Managed Data Pipelines and Tableflow which both empower customers to automate complex data processing all from the WarpStream Console. These products are great, but without a built-in observability feature like Events, customers who want to introspect one of these processes have to fall back to an external tool, and switching from the WarpStream console adds friction to their troubleshooting workflows.

We considered deploying an open-source observability stack alongside each WarpStream cluster, but that would undermine one of WarpStream’s core strengths: no additional infrastructure dependencies. WarpStream is cheap and easy to operate precisely because it’s built on object storage with no extra systems to manage. Adding a sidecar database or log aggregation pipeline would add operational burden and cost.

So we decided to build it directly into WarpStream. WarpStream already has a robust storage engine, so the only missing piece was a small, native query engine. Luckily, many WarpStream engineers helped build Husky at Datadog so we know a little something about building query engines!

This post will have plenty of technical details, but let’s start by diving into the experience of using Events first.

An Intuitive Addition to Your Tool Belt

Events is a built-in observability layer capturing structured operational data from your WarpStream Agents so you can search and visualize it directly in the Console. No external infrastructure is required: the Events data is simply stored as Kafka topics using the same BYOC architecture as WarpStream’s original streaming product. Here’s a quick example of how you might use it.

Concrete Example: Debugging Iceberg Ingestion in Just a Couple Clicks

Suppose you’ve just configured a Warpstream Tableflow cluster to replicate an ‘Orders’ Kafka topic into an Iceberg table, with an AWS Glue catalog integration so your analytics team can query the Iceberg tables data from Athena (AWS’s serverless SQL query engine). A few hours in, you check the WarpStream Console and everything looks healthy. Kafka records are being ingested and the Iceberg table is growing. But when your analysts open Athena, the table isn’t there.

You navigate to the Tableflow cluster in the Console and scroll down to the Events Explorer at the bottom of the table’s detail view. You search for errors: data.log_level == “error”.

Alongside the healthy ingestion events, a parallel stream of aws_glue_table_management_job_failed events appears, one every few minutes since ingestion started. You expand one of the events. The payload includes the table name, the Glue database, and the error message:

"AccessDeniedException: User is not authorized to perform glue:CreateTable on resource"

The IAM role attached to your Agents has the right S3 permissions for object storage, which is why ingestion is working, but is missing the Glue permissions needed to register the table in the catalog. You update the IAM policy, and within minutes the errors are replaced by an aws_glue_table_created event. Your analysts refresh Athena and the table appears.

The data was safe in object storage the entire time, the Iceberg table was healthy, only the catalog registration was failing. Without Events, you would have seen a working pipeline on one side and an empty Athena catalog on the other, with no indication of what was wrong in between. The event payload pointed you directly to the missing permission.

Using the Events Explorer

Events are consumable through the Kafka protocol like any other topic, but raw kafka-console-consumer output isn’t the most pleasant debugging experience. The Events Explorer in the WarpStream Console provides a purpose-built interface for exploring, filtering, and aggregating your events.

The top of the Explorer has four inputs:

1. Event type: which logs to search: Agent logs, ACL logs, Pipeline logs, or all.
2. Time range: quick presets like 15m, 1h, 6h, or custom durations. Absolute date ranges are also supported. The search window cannot exceed 24 hours.
3. Filter: conditions on event fields using a straightforward query language.
4. Sort order: newest first, oldest first, or unsorted.

Results come back as expandable event cards showing the timestamp, type, and message. Expand a card to see the full structured JSON payload. Following the CloudEvents format, application data lives under data.*.

The timeline charts event volume over time, making it easy to spot patterns, for example an error spike after a deploy, periodic ACL denials, a gradual uptick in pipeline failures. You can group by any field in each event’s payload. Group Agent logs by data.log_level to see how the error-to-warning ratio shifts over time, or group ACL events by data.kafka_principal to see which service accounts generate the most denials.

You’ll also find Events Explorer widgets embedded in the ACLs and Pipelines tabs. These provide tightly scoped views relevant to the current context. For example, the ACLs widget pre-filters for ACL events, and the Pipelines widget only shows events generated by the current pipeline.

A Light Footprint

Your Storage, Your Data

WarpStream Agents run in your cloud account and read/write directly to your object storage. Events fit right into this model. Event data is stored as internal Kafka topics, subject to the same encryption, retention policies, and access controls (ACLs) as any other topic. Importantly, Events queries run in the Agents directly, just like Kafka Produce and Fetch requests, so you don’t have to pay for large volumes of data to be egressed from your VPC.

Query results do pass through the control plane temporarily so they can be rendered in the WarpStream Console, but they aren’t persisted anywhere. In addition, the Events topics themselves are hard-coded to only contain operational metadata such as Agent logs, request types, ACL decisions, and Agent diagnostics. They never contain your topics’ actual messages or raw data.

Cost Impact

Events contribute to your storage and API costs just like any other topic data persisted in your object storage bucket, but we’ve specifically tuned Events to be cheap. For moderately sized clusters, the expected impact is less than a few dollars per month.

If cost is a concern, i.e. for very high-throughput clusters, you can selectively disable event types you don’t need, for example, keeping ACL logs and turning off Agent logs. You can also reduce each event type’s retention period below the 6-hour default.

But How Does It Work? The Query Engine

To put it bluntly, we bolted a query engine onto a distributed log that stores data in a row-oriented format. The storage layer is not columnar, which means we’re never going to win any benchmark competitions. But that’s okay. Our Events product doesn’t need to be the fastest on the market. It just needs to support infinite cardinality and be fast enough, cheap enough, and easy enough to use that it makes life easier for our customers. And that’s what we built.

Lifecycle of a Query

When you submit a query through the Events Explorer, it gets routed to one of your Agents as a query job. The Agent then:

1. Parses the query into an Abstract Syntax Tree (AST).
2. Compiles the AST into a logical plan (filter, project, aggregate, sort, limit nodes).
3. Physically plans the query by resolving topic metadata: fetching partition counts, start/end offsets, and using topic metadata to narrow the offset ranges based on the time filter.
4. Splits the offset ranges into tasks. Each task covers a contiguous range of offsets for a single partition.
5. Schedules tasks for execution in stages, with results flowing between stages via output buffers.
6. Executes tasks using our in-house vectorized query engine.

For now, a single Agent executes the entire query, though the architecture is designed to distribute tasks across multiple Agents in the future.

Pruning Timestamps

The core challenge is that queries are scoped to a time range, but data in Kafka is organized by offset, not timestamp. While WarpStream supports ListOffsets for coarse time-to-offset translation, the index is approximative (for performance reasons), and small time windows like “the last hour” can still end up scanning much more data than necessary.

The query engine addresses this with progressive timestamp pruning. As tasks complete, the engine records the actual timestamp ranges observed at each offset range. We call these timestamp watermarks. These watermarks are then used to skip pending tasks whose offset ranges fall entirely outside the query’s time filter.

The pruning works in both directions:

• Lower offsets: If a completed task at offset 200 has a minimum timestamp of 1:30 AM, and the query filters for 2:00 AM–4:00 AM, then all tasks with offsets below 200 can be safely skipped: their timestamps can only be earlier.
• Higher offsets: Similarly, if a completed task shows timestamps already past the query’s end time, all tasks at higher offsets can be skipped.

To maximize pruning effectiveness, tasks are not scheduled sequentially. Instead, the scheduler uses a golden-ratio-based spacing strategy (similar to Kronecker search and golden section search) to spread early tasks across the offset space, sampling from the middle first and then filling in gaps. This maximizes the chances that the first few completed tasks produce watermarks that eliminate large swaths of remaining work.

On a typical narrow time-range query, this pruning eliminates the majority of scheduled tasks and allows us to avoid scanning all the stored data.

The Direct Kafka Client

The query engine reads data using the Kafka protocol, fetching records at specific offset ranges just like a consumer would. But in the normal Kafka path, data flows through a chain of processing: the Agent handling the fetch reads from blob storage (or cache), decompresses the data, wraps it in Kafka fetch responses, compresses it for network transfer, and sends it to the requesting Agent, which then decompresses it again (learn about how we handle distributed file caches in this blog post). Even when the query is running on an Agent that is capable of serving the fetch request directly, this involves real network I/O and redundant compression cycles.

The query engine short-circuits this with a direct in-memory client. It connects to itself using Go’s net.Pipe(), creating an in-memory bidirectional pipe that looks like a network connection to both ends but never hits the network stack. On top of that, the direct client signals via its client ID that compression should be disabled, eliminating the compress-decompress round entirely. Additionally, this ensures that the Events feature always works, even when the cluster is secured with advanced authentication schemes like mTLS.

These two optimizations–in-memory transport and disabled compression–more than doubled the data read throughput of the query engine in our benchmarks. Is it faster than a purpose-built observability solution? Absolutely not, but it’s cheap, easy to use, adds zero additional dependencies, and is integrated natively into the product.

Query Fairness and Protecting Ingestion

Events is designed as an occasional debugging tool, not a primary observability system. To make sure queries never impact live Kafka workloads, several safeguards are in place:

• Memory limits: Configurable caps on how much memory a single query can consume.
• Concurrency control: A semaphore in the control plane limits the maximum number of concurrent queries to 2 per cluster, regardless of the number of Agents. This is intentionally conservative for now and will be relaxed as the system matures.
• Scan limits: Restrictions on the amount of data scanned from Kafka per query, to minimize pressure on Agents handling fetch requests.
• Query only Agents: It’s possible to restrict some Agents to query workloads (see the documentation here).

More Optimizations

Beyond pruning and the direct client, the query engine applies several standard techniques:

• Metadata-only evaluation: For queries that only need record metadata (e.g., counting events by timestamp), the engine skips decoding the record value entirely.
• Early exit: For list-events and TopN queries, scanning stops as soon as enough results have been collected.
• Adaptive fetch sizing: List-like queries use smaller fetches (to minimize over-reading), while aggregate queries use larger fetches (to maximize throughput).
• Progressive results: For timeline queries, multiple sub-queries are scheduled to show results progressively for a more interactive UI.

Data Streams and Future Plans

Events launches with three data streams:

1. Agent logs: structured logs from every Agent in the cluster, regardless of role. Filter by log level, search for specific error messages, or correlate Agent behavior with a timestamp.
2. ACL events: every authorization decision, including denials. Captures the principal, resource, operation, and reason. Useful for rolling out ACL changes, managing multi-tenant clusters, and auditing shadow ACL decisions.
3. Pipeline events: execution logs from WarpStream Managed Data Pipelines. These help you understand why a pipeline is failing and make the Tableflow product much easier to operate, since you can see processing feedback directly in the Console without context-switching to an external logging system.

We plan to add new data streams over time as we identify more areas where built-in observability can make our customers’ lives easier.

Audit Logs

The same infrastructure that powers Events also drives WarpStream’s Audit Logs feature. Audit Logs track control plane and data plane actions–Console requests, API calls, Kafka authentication/authorization events, and Schema Registry operations–using the same CloudEvents format. They are queryable through the Events Explorer with the same query language and enjoy the same query engine optimizations.

The only difference is that in the audit logs product, the WarpStream control plane hosts the Kafka topics and query engine because many audit log events are not tied to any specific cluster.

Getting Started

Events are available now for all WarpStream clusters. To enable them, you’ll need to upgrade your Agents to version v770 or later. Once your Agents are updated:

1. Go to your cluster’s Events tab in the Console.
2. Turn on Global Events Status.
3. Optionally toggle individual event types on or off.
4. Wait a few minutes for your Agents to capture their first events.
5. Scroll down to the Events Explorer and run a search.

The global status and individual event types can be toggled on and off via the API or the Terraform provider.

Event topics have a default retention of 6 hours. Retention and partition count are configurable per event type. Find these details and more in the Events Explorer’s public documentation page.

When teams run Kafka at scale, “who did what, and when?” becomes a vital question that needs an answer. A topic gets deleted, a consumer group disappears, or a new ACL shows up, and nobody knows who made the change. In regulated industries, that gap isn’t just inconvenient, it can be a compliance violation. Even outside of compliance, the absence of a reliable audit trail makes incident response slower and root-cause analysis harder.

Today we’re releasing Audit Logs for WarpStream, giving you a complete, structured record of every authentication action, authorization decision, and platform operation across your clusters.

Why Audit Logs?

Most Kafka operators are familiar with the pain. Kafka itself provides limited built-in auditing. You can piece together some information from broker logs, but those logs are unstructured, scattered across brokers, and not designed for long-term retention or compliance. Building a proper audit trail on top of vanilla Kafka typically means stitching together log aggregation pipelines, parsing free-text log lines, and hoping nothing falls through the cracks.

With WarpStream, because all metadata flows through the hosted control plane, we can expose a centralized view of every operation that occurs across your clusters. The logs are structured, queryable, and consumable using the tools you already know.

What Gets Logged?

Audit Logs capture two broad categories of events:

1. Cluster Audit Logs track Kafka-level operations on all Pro and Enterprise clusters. Every authentication attempt, every authorization decision, and every administrative action, including topic creation and deletion, consumer group deletion, ACL modification, and more, is recorded as a structured event. If a client connects with the wrong credentials and gets rejected, an audit log will be emitted. If someone creates a topic, you will see the full configuration that was applied.
2. Platform Audit Logs track account-level operations in the WarpStream Console. Creating or deleting clusters, managing API keys, modifying user accounts — any mutating operation against the WarpStream API generates an audit event.

Every event follows the CloudEvents spec and conforms to the same schema used by Confluent Cloud Audit Logs, so if you already have tooling built around that format, it works out of the box.

Built on Kafka, Consumed With Kafka

WarpStream Audit Logs aren’t just written to a database somewhere. They’re produced into a fully-managed WarpStream cluster running on WarpStream’s own cloud infrastructure. That means you can consume your Audit Logs using the Kafka protocol, with any Kafka client, and export them anywhere: your SIEM, a data lake, an Iceberg table via Tableflow, and more.

This approach gives you the best of both worlds. You get a built-in UI in the WarpStream Console for browsing and searching audit events directly, and you also get the full power and flexibility of the Kafka ecosystem for building whatever downstream integrations your compliance or security team needs.

To get started consuming Audit Logs programmatically, just navigate to the Audit section of the Console, create SASL credentials under the “Credentials” tab, and use the connection details from the “Connect” tab. It’s the same workflow you’d use to connect any Kafka client to any WarpStream cluster.

What Does an Audit Log Event Look Like?

Each audit log event is a structured JSON payload. Here’s a simplified example of what you’d see when a client creates a topic:

{
  "type": "warpstream.kafka/request",
  "time": "2025-12-17T09:29:36.120450058Z",
  "data": {
    "methodName": "kafka.CreateTopics",
    "authenticationInfo": {
      "credentials": {
        "mechanism": "SASL_SCRAM",
        "idSecretCredentials": {
          "credentialId": "ccun_xxxx"
        }
      }
    },
    "request": {
      "accessType": "MODIFICATION",
      "data": {
        "name": "my-new-topic",
        "numPartitions": 4,
        "replicationFactor": 1,
        "configs": [
          {"name": "retention.ms", "value": "86400000"},
          {"name": "cleanup.policy", "value": "compact,delete"}
        ]
      }
    },
    "result": {
      "status": "SUCCESS",
      "data": {"errorCode": 0}
    }
  }
}

The event tells you exactly what happened (a topic was created), who did it (which credential was used), when it happened, and what the outcome was. Failed operations are logged, too. If a DeleteGroups call fails because the group doesn’t exist, you’ll see the error code and message in the result, which is invaluable for debugging client issues or spotting misconfigured applications.

Platform-level events follow the same structure.

Note that the Audit Logs events still only contain metadata. No data is egressed from your WarpStream clusters, and no access is delegated to WarpStream. The hosted cluster is simply a convenient interface that we expose so that you don’t have to deploy and manage another BYOC cluster in order to store and retrieve your Audit Logs.

Pricing

Because the Audit Logs cluster is hosted on WarpStream’s infrastructure rather than in your cloud account, billing works slightly differently than for BYOC clusters. It’s metered across four dimensions: uncompressed data written, storage (GiB-minutes), network ingress, and network egress.

For the vast majority of use cases, Audit Logs are inexpensive. Most clusters produce well under 500 GiB/month of audit log data, which translates to less than $10/month for writes. Storage and networking charges are similarly modest. You can find the full pricing breakdown in our docs.

Getting Started

Audit Logs are available today for all WarpStream accounts on Pro or Enterprise clusters. To enable them:

1. Make sure your Agents are running version v736 or above.
2. Navigate to the “Audit” section in the WarpStream Console.
3. Enable Audit Logs with a single click.

Once enabled, logs start flowing immediately. No additional configuration, sidecar processes, or log aggregation pipelines need to be built.