WarpStream Newsletter #4: Data Pipelines, Zero Disks, BYOC and More
Tags: dataengineering, apachekafka, datastreaming, warpstream
Published: 2024-07-15 14:09:42

---
title: WarpStream Newsletter #4: Data Pipelines, Zero Disks, BYOC and More
published: true
date: 2024-07-10 17:20:40 UTC
tags: dataengineering,apachekafka,datastreaming,warpstream
canonical_url: 
---

Welcome to the fourth issue of the WarpStream newsletter. A lot has happened since our last newsletter: we’ve released five new blogs, made a bunch of product updates, and added new social channels (like [Facebook](https://www.facebook.com/warpstream) and [YouTube](https://www.youtube.com/@warpstreamlabs). Connect with us on social media and other platforms to stay updated via the links in the social footer at the bottom of this email.

### Lots of New Blog Posts

### [Introducing WarpStream Managed Data Pipelines for BYOC clusters](https://www.warpstream.com/blog/introducing-warpstream-managed-data-pipelines-for-byoc-clusters)

For WarpStream BYOC clusters, Managed Data Pipelines provide a fully-managed SaaS user experience for [Bento](https://warpstreamlabs.github.io/bento/), a lightweight stream processing framework that offers much of the functionality of Kafka Connect, without sacrificing any of the cost benefits, data sovereignty, or deployment flexibility of the BYOC deployment model and comes with version control.

![](https://cdn-images-1.medium.com/max/1024/0*xFIGBzfM7w1KJSWm.png)

### [Pixel Federation Powers Mobile Analytics Platform with WarpStream, saves 83% over MSK](https://www.warpstream.com/blog/pixelfederation-powers-mobile-analytics-platform-with-warpstream)

Pixel Federation’s mobile games have millions of users, so you can imagine how many events and Kafka topics they have. By swapping MSK for WarpStream, they not only drastically reduced their costs, but were able to ditch complex VPC peering in favor of simpler agent groups.

![](https://cdn-images-1.medium.com/max/1024/0*tMPR-Mf4pImJDw0J.png)

**Interested in Learning More About WarpStream?**

[**Book a call**](https://calendly.com/d/3x5-79z-zc6/warpstream-demo-45-minutes)

### [Zero Disks is Better (for Kafka)](https://www.warpstream.com/blog/zero-disks-is-better-for-kafka)

In a prior blog, we discussed how [tiered storage won’t fix Kafka](https://www.warpstream.com/blog/tiered-storage-wont-fix-kafka). The end goal is not some disks but zero disks. We cover how WarpStream’s Zero Disk Architecture (ZDA) allows you to do things like trivial or dead-simple auto-scaling of Kafka brokers (“agents” in WarpStream terminology), isolate workloads with agent groups, and easily run your entire data pipeline in your virtual private cloud (VPC) without the need for custom code or additional services.

![](https://cdn-images-1.medium.com/max/1024/0*XMsemAcHrhpXUBiW.png)

### [Secure by default: How WarpStream’s BYOC deployment model secures the most sensitive workloads](https://www.warpstream.com/blog/secure-by-default-how-warpstreams-byoc-deployment-model-secures-the-most-sensitive-workloads)

WarpStream’s BYOC model is a hybrid approach that balances the two common cloud deployment models (fully self-managed and fully hosted SaaS). By splitting the software into discrete data and control planes, it ensures data privacy and sovereignty, compliance, cost optimization, and control.

![](https://cdn-images-1.medium.com/max/1024/0*reNkpam00xifWR6F.png)

**Try WarpStream With $400 in Free Credits**

[**Get Started For Free**](https://console.warpstream.com/signup)

### [Multiple Regions, Single Pane of Glass](https://www.warpstream.com/blog/multiple-regions-single-pane-of-glass)

A common problem when building infrastructure-as-a-service products is the need to provide highly available and isolated resources in many different regions while also having the overall product present as a “single pane of glass” to end-users. We review the options available to solve this and what we ultimately used (pushed-based replication).

![](https://cdn-images-1.medium.com/max/1024/0*WAIyN6yCDVuIKKnj.png)

### Recent Product Updates

### [Managed Data Pipelines](https://docs.warpstream.com/warpstream/configuration/bento)

BYOC customers can now use Managed Data Pipelines. These combine the power of WarpStream’s control plane with [Bento](https://warpstreamlabs.github.io/bento/), an open-source streaming processing platform.

This provides much of the same functionality as Kafka Connect and additional stream processing functionality like single message transforms, aggregations, multiplexing, enrichments, and native support for WebAssembly (WASM).

Pipelines run in your VPC and on your VMs, and data is processed in your buckets. WarpStream has zero access to this data. WarpStream provides a helpful UI for creating and editing pipelines, the ability to pause and resume pipelines dynamically, and version control.

### [Lots of New Metrics](https://docs.warpstream.com/warpstream/overview/change-log)

We’ve added new metrics (and deprecated unnecessary ones) with nearly every release. We’ve recapped some of these new metrics below. You can check out [our official changelog](https://docs.warpstream.com/warpstream/overview/change-log) to get the full list.

- **warpstream\_consumer\_group\_generation\_id** = This metric indicates the generation number of the consumer group, incrementing by one with each rebalance. It serves as an effective indicator for detecting occurrences of rebalances.
- **warpstream\_agent\_kafka\_fetch\_uncompressed\_bytes** = Tracks the total uncompressed bytes fetched, replacing warpstream\_agent\_kafka\_fetch\_bytes\_sent metric.
- **warpstream\_consumer\_group\_generation\_id** = Uses the consumer\_group tag. This metric indicates the generation number of the consumer group, incrementing by one with each rebalance. It serves as an effective indicator for detecting occurrences of rebalances.

### Coming Soon: Kafka Transactions

As we announced in our previous newsletter, the team is working on building in support for Kafka Transactions and expects to finish this work soon. If you want to use WarpStream for a workload requiring Transactions, please [contact us](https://www.warpstream.com/contact-us)! We would love to chat.

[**Try WarpStream With $400 in Free Credits**](https://console.warpstream.com/signup)

WarpStream is free to try. After you create your account, it will be loaded with $400 in free credits so you can test how easy it is to set up and use WarpStream.

[**Get Started For Free**](https://campaigns-events.was-1.sendpdr.com/track/link/v2_y10xn4ev41/9q67kt2pd8uylyuzfmulr7km0/v2_2nyq2boq67)