GCP Dataproc Serverless cost: paying for Spark without managing clusters

Quick answer

Dataproc Serverless runs Spark jobs on demand without a standing cluster, billing per Data Compute Unit (DCU) for the job's duration, plus memory and shuffle storage, so you pay only while a job runs and nothing between jobs. This eliminates the idle-cluster cost that a managed Dataproc cluster incurs when it sits waiting for work. Serverless typically beats a standing cluster when your Spark jobs are intermittent or bursty, since a managed cluster left running between jobs pays for idle capacity, while a busy cluster running near-constantly can be cheaper on committed or sustained-discounted Compute Engine. The core tradeoff is idle cost versus utilization.

Running Spark on Dataproc has traditionally meant standing up a cluster, and a cluster costs money whenever it is running, whether or not a job is executing. Dataproc Serverless removes the cluster from the equation: you submit a Spark job, Google provisions compute for its duration, and you pay only for that. The cost question becomes whether serverless per-job billing or a standing cluster's utilization economics wins for your workload.

How Serverless bills

Component	Billing
Compute	Per Data Compute Unit (DCU) for job duration
Memory	Per GB for job duration
Shuffle storage	Per GB of shuffle data
Idle between jobs	Nothing

Dataproc Serverless bills per Data Compute Unit for the compute a job uses, plus memory and shuffle storage, all measured for the duration of the job. When no job is running, nothing is billed. This contrasts with a managed Dataproc cluster, which bills the underlying Compute Engine VMs plus a small Dataproc premium per vCPU-hour for as long as the cluster runs, including the time it sits idle waiting for the next job.

Why idle cost is the deciding factor

A standing Dataproc cluster is efficient only when it is busy. If jobs run a few hours a day and the cluster stays up around the clock, most of what you pay is idle capacity. Serverless flips that: with no cluster to keep warm, intermittent and bursty workloads pay only for the minutes their jobs run, which can be dramatically cheaper than a cluster idling between jobs. The more sporadic your Spark usage, the more serverless saves by eliminating idle time.

When a standing cluster still wins

A cluster running near-constantly, with jobs back to back, has little idle time to eliminate, so serverless loses its main advantage. At high sustained utilization, a managed cluster on Compute Engine can be cheaper because it benefits from sustained use discounts, committed use discounts, and Spot VMs for the workers, which drive the per-hour compute cost below the serverless per-DCU rate. So very busy, steady Spark workloads may favor a well-utilized standing cluster with Spot workers.

Choosing the model

Look at your Spark job pattern. For intermittent, bursty, or unpredictable jobs, Dataproc Serverless usually wins by charging only for job duration and nothing for idle time. For steady, near-constant workloads, a managed cluster with Spot workers and committed or sustained discounts can be cheaper thanks to high utilization. Ephemeral per-job clusters that spin up and tear down are a middle option. Price both Dataproc Serverless and a managed cluster for your workload against the resource catalog so the cheaper model is chosen before the pipeline runs.

FAQ

How is Dataproc Serverless priced?

It bills per Data Compute Unit for the compute a Spark job uses, plus memory and shuffle storage, all measured for the job's duration. When no job runs, nothing is billed. This differs from a managed Dataproc cluster, which bills the underlying Compute Engine VMs plus a small Dataproc premium for as long as the cluster runs, including idle time between jobs.

When is Dataproc Serverless cheaper than a cluster?

When your Spark jobs are intermittent or bursty. A standing cluster left running between jobs pays for idle capacity, while serverless charges only for the minutes each job runs and nothing between jobs. The more sporadic your Spark usage, the more serverless saves by eliminating idle time, which is often dramatic for jobs that run a few hours a day.

When does a standing Dataproc cluster win on cost?

When it runs near-constantly with jobs back to back, so there is little idle time to eliminate. At high sustained utilization, a managed cluster on Compute Engine benefits from sustained use discounts, committed use discounts, and Spot VMs for workers, which can drive the per-hour compute cost below the serverless per-DCU rate. Very busy, steady Spark workloads may favor a well-utilized cluster.

What is a Data Compute Unit in Dataproc Serverless?

A Data Compute Unit (DCU) is the metering unit for compute in Dataproc Serverless, billed for the duration of a Spark job. Your bill is the DCUs the job consumes plus memory and shuffle storage over the job's runtime. Because it is measured per job rather than per cluster-hour, you pay only for actual job execution, not for idle cluster time.

How does C3X help with Dataproc cost?

C3X prices your Dataproc infrastructure from Terraform before you deploy, so you can compare Dataproc Serverless against a managed cluster for your Spark workload at design time. That helps you weigh the idle-cost elimination of serverless against the high-utilization discounts of a standing cluster with Spot workers, choosing the cheaper model before the pipeline runs rather than after the bill arrives.

What to do next

Compare serverless and managed Spark on cost before you run. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.