GCP Dataflow cost: workers, memory, and the shuffle charge

Quick answer

Dataflow charges per vCPU-hour and GB-hour of worker time, plus a data-processed charge for Dataflow Shuffle (batch) or Streaming Engine (streaming) that moves state off the workers. Right-size worker machine types, cap autoscaling, and let the managed shuffle shrink workers to control the bill.

Dataflow runs Apache Beam pipelines as a managed service. Its cost is worker compute plus a charge for the data-shuffling machinery, and how you configure workers and shuffle decides the total.

Worker compute

You pay per vCPU-hour and per GB-hour for the workers that run your pipeline, based on their machine type and how long the job runs. Autoscaling adjusts worker count to the work, so a job's compute cost is roughly machine size times worker-hours.

Shuffle and Streaming Engine

Dimension	Applies to
vCPU-hour + GB-hour	All jobs (worker compute)
Dataflow Shuffle (data processed)	Batch jobs
Streaming Engine (data processed)	Streaming jobs

Dataflow Shuffle (batch) and Streaming Engine (streaming) move shuffle and state off the worker VMs into a managed service, billed per gigabyte processed. This adds a line but lets workers be smaller and autoscale better, often reducing total cost despite the extra charge.

Reducing Dataflow cost

Right-size worker machine types to the pipeline, cap the maximum number of workers so autoscaling cannot run away, and prefer the managed shuffle so workers stay lean. Reduce data processed by filtering early in the pipeline, and use Spot (preemptible) workers for fault-tolerant batch jobs, the same preemptible saving as elsewhere on GCP.

FAQ

How is GCP Dataflow priced?

Per vCPU-hour and GB-hour of worker time, plus a data-processed charge for Dataflow Shuffle on batch jobs or Streaming Engine on streaming jobs. Worker compute plus the shuffle charge make up the bill.

What is the Dataflow Shuffle charge?

A per-gigabyte charge for the managed service that handles data shuffling off the worker VMs on batch jobs (Streaming Engine is the streaming equivalent). It adds a line but lets workers be smaller and autoscale better, often lowering total cost.

How do I reduce Dataflow cost?

Right-size worker machine types, cap the maximum worker count so autoscaling cannot run away, use the managed shuffle so workers stay lean, filter data early to reduce processing, and use preemptible workers for fault-tolerant batch jobs.

Do preemptible workers save money on Dataflow?

Yes, for fault-tolerant batch jobs. Preemptible (Spot) VMs cost far less per hour, and Dataflow can use them for workers. Because batch work reschedules when a worker is reclaimed, the risk is low and the saving is significant.

Does Streaming Engine save money?

Often, despite adding a data-processed charge. Streaming Engine moves state off the workers, so they can be smaller and scale more responsively, which typically reduces worker compute enough to more than offset the Streaming Engine charge.

Does C3X estimate Dataflow cost?

Dataflow cost is driven by worker-hours and data processed, which are runtime usage inputs. C3X prices the surrounding infrastructure, and you model worker size, runtime, and data volume to estimate the job cost.

What to do next

Estimate the infrastructure around your pipelines before you run them. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.