AzureAzure HDInsightAnalytics

`azurerm_hdinsight_kafka_cluster` cost estimation

A managed Apache Kafka cluster on HDInsight, billed per node-hour for head and worker (broker) VMs. A 2-head + 3-broker cluster is ~$1,894/month, running continuously.

An azurerm_hdinsight_kafka_cluster runs managed Apache Kafka on Azure HDInsight. Cost is per node-hour across all nodes — head nodes plus the worker (broker) nodes — at the VM rate plus HDInsight surcharge. A 2-head + 3-broker D12v2-class cluster is ~$0.519/node-hour × 5 × 730 ≈ $1,894/month, billed continuously.

Unlike batch analytics clusters, a Kafka cluster is inherently always-on — it's a messaging backbone, so you can't just create-and-delete it per job. That makes node count and node size the primary levers: provision brokers for your throughput and retention needs (Kafka storage is on managed disks attached to brokers), and right-size the VM class to the message volume.

Because it must run continuously, this is a case where alternatives matter for cost: Azure Event Hubs (which offers a Kafka-compatible endpoint) is serverless/throughput-billed and often far cheaper than running a standing HDInsight Kafka cluster, unless you specifically need full self-managed Kafka. Reserved VM instances discount the always-on broker nodes.

c3x prices the cluster from the worker (broker) count and node size, so the always-on cost is visible before deployment.

Terraform example

A minimal but realistic configuration that C3X can estimate.

resource "azurerm_hdinsight_kafka_cluster" "events" {
  name                = "kafka-cluster"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  cluster_version     = "5.1"
  tier                = "Standard"

  roles {
    head_node {
      vm_size = "Standard_D12_v2"
      # ... credentials
    }
    worker_node {
      vm_size                  = "Standard_D12_v2"
      target_instance_count    = 3
      number_of_disks_per_node = 2
    }
    zookeeper_node {
      vm_size = "Standard_A2_v2"
    }
  }
}

Pricing dimensions

What you actually pay for when you provision azurerm_hdinsight_kafka_cluster.

Dimension	Unit	What's being charged
Cluster nodes	per node-hour	All nodes (2 head + brokers) bill per node-hour at the VM rate plus HDInsight surcharge, continuously. Kafka clusters run 24/7. $0.519/node-hour (D12v2-class) → 5 nodes ≈ $1,894.35/month
Broker disks	per disk-month	Managed disks attached to brokers hold Kafka log data, billed per disk on top of nodes.

Sample C3X output

2 head + 3 broker nodes (D12v2-class), running 24/7:

azurerm_hdinsight_kafka_cluster.events
└─ Cluster nodes (5 × D12v2-class)   3650 node-hours   $1,894.35
                                     Monthly           $1,894.35

Optimization tips

Common ways to reduce azurerm_hdinsight_kafka_cluster cost without changing the workload.

Consider Event Hubs with the Kafka endpoint

Large vs an always-on cluster for many workloads

Azure Event Hubs offers a Kafka-compatible endpoint and is serverless/throughput-billed — often far cheaper than running a standing HDInsight Kafka cluster, unless you specifically need full self-managed Kafka with custom configs.

Right-size broker count and VM size

Proportional to right-sizing

Cost scales with broker count and node size. Provision brokers for your real throughput and retention (disk) needs, and match the VM class to message volume rather than over-provisioning a standing cluster.

Reserve the broker nodes

40–60% on the steady cluster

A Kafka cluster runs continuously, so the broker VMs are an ideal reservation target — a 1-3 year reservation discounts the always-on nodes significantly.

Tune retention to control broker disk cost

Per disk-month reduced

Kafka log retention sets how much disk the brokers need. Retaining only what consumers require (rather than weeks of logs) reduces the managed-disk cost attached to brokers.

FAQ

How is an HDInsight Kafka cluster billed?

Per node-hour across all nodes (head + brokers) at the VM rate plus HDInsight surcharge, continuously, plus managed disks on brokers for Kafka log storage. A 2-head + 3-broker D12v2 cluster is ~$1,894/month. Kafka clusters are always-on, so the cost is steady.

Is there a cheaper alternative to HDInsight Kafka?

Often yes — Azure Event Hubs offers a Kafka-compatible endpoint and bills by throughput rather than standing cluster nodes, which is far cheaper for many workloads. Use HDInsight Kafka when you need full self-managed Kafka; otherwise compare Event Hubs.