Kafka cost optimization: brokers, storage, and managed vs self-hosted

Quick answer

Kafka cost is the broker compute (instances running the cluster), the storage of retained messages, cross-AZ replication transfer (replicating partitions across zones is billed data transfer), and, on a managed service (MSK, Confluent), the service premium. The levers: set retention to what consumers actually need (long retention multiplies storage), right-size brokers and partitions, use tiered storage to offload old data to cheap object storage, and minimize cross-AZ transfer. Retention and replication are the underestimated costs.

Apache Kafka underpins many streaming architectures, and its cost has several parts that are easy to underestimate, especially message retention and cross-zone replication. Whether self-hosted or on a managed service, the drivers are broker compute, retained-message storage, replication transfer, and the managed premium if any.

Where Kafka cost comes from

Component	Cost driver
Broker compute	Instances running the cluster
Storage	Retained messages (retention x throughput)
Cross-AZ replication	Data transfer replicating partitions
Managed premium	MSK/Confluent service fee

Broker compute runs the cluster continuously. Storage holds retained messages, and long retention on high-throughput topics stores enormous volumes. Replicating partitions across availability zones for durability generates cross-AZ data transfer, a real and often-overlooked cost. Managed services (MSK, Confluent) add a premium over self-hosting for operations.

Retention is a storage multiplier

Retention is the sneaky cost: a high-throughput topic retained for weeks stores far more than one retained for days, multiplying storage. Set retention to what consumers actually need to re-read, not a default long window, and use tiered storage (offloading older log segments to cheap object storage like S3) where the platform supports it, so hot broker storage holds only recent data. This can cut storage cost dramatically.

Managed vs self-hosted

Managed Kafka (MSK, Confluent Cloud) removes operational burden at a premium; self-hosted is cheaper per unit but you run and scale it, a real operational cost. For most teams the managed premium is worth it versus the burden of operating Kafka; at very large scale or with strong platform teams, self-hosted can be cheaper. Either way the retention, replication, and right-sizing levers apply.

Controlling Kafka cost

Set retention to actual consumer needs (the biggest underestimated cost), use tiered storage to offload old segments to cheap object storage, right-size brokers and partition counts to throughput, minimize cross-AZ replication transfer where durability requirements allow, and weigh managed premium against operational cost. Because retention-driven storage and replication transfer are the hidden costs, targeting those alongside right-sized brokers governs the bill, and consider whether streaming is even needed versus batch.

FAQ

What drives Kafka cost?

The broker compute (instances running the cluster), the storage of retained messages (retention times throughput), cross-AZ replication transfer (replicating partitions across zones is billed data transfer), and, on a managed service like MSK or Confluent, the service premium. Retention-driven storage and cross-AZ replication are the most underestimated costs.

How do I reduce Kafka cost?

Set retention to what consumers actually need (long retention multiplies storage), use tiered storage to offload old log segments to cheap object storage, right-size brokers and partition counts to throughput, minimize cross-AZ replication transfer where durability allows, and weigh the managed premium against operational cost. Retention and replication are the biggest hidden costs to target.

Why is Kafka retention a significant cost?

Because storage scales with retention times throughput, a high-throughput topic retained for weeks stores far more data than one retained for days, multiplying storage cost. Long default retention windows on busy topics quietly accumulate large storage. Setting retention to what consumers actually need to re-read, rather than a long default, is a major saving.

What is tiered storage in Kafka and does it save money?

Tiered storage offloads older log segments from the brokers' fast local storage to cheap object storage (like S3), so hot broker storage holds only recent data while older retained data lives cheaply. Where the platform supports it, this dramatically cuts storage cost for topics with long retention, since most retained data is old and rarely re-read.

Is managed Kafka or self-hosted cheaper?

Managed Kafka (MSK, Confluent Cloud) adds a premium but removes the significant operational burden of running and scaling Kafka. For most teams the managed premium is worth it versus that burden. At very large scale or with strong platform teams, self-hosted can be cheaper per unit. Either way, retention, replication, and right-sizing levers apply.

Does C3X estimate Kafka cost?

C3X prices the Kafka infrastructure (broker instances or managed cluster, storage) from your Terraform, so the base cost appears before deploy. Throughput, retention, and cross-AZ replication volume are usage inputs you model to estimate the storage and transfer costs that often dominate.

What to do next

Price your Kafka cluster before you deploy. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.