Job clusters vs all purpose clusters: a 3.7x cost difference for identical work
The same notebook, the same data, the same runtime, and a bill that differs by a factor of nearly four. The cluster type you attach a scheduled job to is the cheapest big win in Databricks.
Quick answer
Job clusters are created for a single run and terminated when it finishes, and they bill at the Jobs Compute DBU rate, around $0.15 per DBU on AWS Premium. All purpose clusters stay alive for interactive use and bill at around $0.55 per DBU, roughly 3.7x more. For identical scheduled work, a job cluster costs about 73 percent less in DBUs and eliminates idle time between runs entirely. The only reasons to keep a scheduled job on an all purpose cluster are sub minute startup requirements or shared session state, and both have better solutions.
Two clusters, same instance types, same worker count, same Spark version, running the same notebook against the same table. One costs $180 a month, the other $660. The difference is a dropdown in the job definition. This is not a subtle optimization requiring a performance engineer; it is a configuration audit that a single person can complete in an afternoon.
What each cluster type is for
| Job cluster | All purpose cluster | |
|---|---|---|
| Lifecycle | Created per run, terminated after | Long lived, manually or auto terminated |
| DBU rate (AWS Premium) | ~$0.15 | ~$0.55 |
| Idle billing | None | Full rate until termination |
| Startup | 2 to 6 minutes | Already warm |
| Shared by users | No | Yes |
| Best for | Scheduled pipelines | Notebook development, exploration |
The pricing reflects what you are actually buying. An all purpose cluster sells availability: it is there, warm, shared, ready the moment someone runs a cell. A job cluster sells execution: it appears, does the work, and disappears. Scheduled pipelines do not need availability, so paying for it is waste.
The arithmetic on a real pipeline
Take a nightly transformation that runs 45 minutes on a cluster consuming 16 DBUs per hour. That is 12 DBUs per run, 360 DBUs a month across 30 runs.
| Configuration | DBU cost/month | VM cost/month | Total |
|---|---|---|---|
| Job cluster, on demand workers | $54 | ~$63 | ~$117 |
| Job cluster, spot workers | $54 | ~$20 | ~$74 |
| All purpose, auto term 60 min | $198 plus idle | ~$147 | ~$345 |
| All purpose, never terminated | ~$6,340 | ~$2,020 | ~$8,360 |
The last row is the one that ruins budgets. A cluster of that size left permanently alive burns 16 DBUs an hour times 720 hours, 11,520 DBUs, at the all purpose rate. Nobody chooses that deliberately; it happens because auto termination was disabled during a debugging session in March and never re enabled.
The idle problem is the bigger half
The DBU rate difference is 3.7x, but the idle difference can be far larger. A job cluster running 45 minutes a night bills 22.5 hours a month. An all purpose cluster with a 60 minute auto termination bills at least 45 hours a month for the same work, double, and that assumes nobody touches it in between. Add a developer who attaches a notebook at 9am and leaves it open through lunch and you are into hundreds of hours. Job clusters make idle billing structurally impossible rather than dependent on a timeout setting being correct.
The objections, and what to do instead
"Startup takes too long." Job cluster startup is typically 2 to 6 minutes. For a nightly batch job that is irrelevant. If the job genuinely needs sub minute start, use a serverless compute option where available, or a cluster pool. Pools keep idle instances warm at cloud provider cost only, with no DBU charge while they sit in the pool, which cuts startup to under a minute while keeping the Jobs Compute rate for the actual run.
"We need the cluster for ad hoc debugging." Keep one small all purpose cluster for that, sized for a person rather than a pipeline, with a 30 minute auto termination. A two worker debugging cluster costs a fraction of the eight worker production shaped cluster teams usually leave running, and separating the two means a debugging session can never accidentally bill at production scale overnight.
"Several jobs share the cluster." Co locating jobs on one all purpose cluster to avoid repeated startup is a false economy at these rates. Run them as tasks within a single job so they share one job cluster, which gets the cheap rate, one startup, and automatic teardown.
"We need the cached state." Cached DataFrames do not survive between scheduled runs in any useful way, and relying on warm cache across runs makes pipelines non reproducible. Persist intermediates to Delta tables instead.
Auditing your workspace
List every scheduled job and record which cluster it attaches to. Anything pointing at an existing all purpose cluster is a candidate. Convert it to a job cluster with the same instance types and worker count, add spot workers with an on demand driver, and confirm runtime is unchanged. Then set auto termination on the remaining development clusters to 30 minutes and remove permission to disable it. Track the result: teams commonly see 40 to 60 percent off total Databricks spend from this audit alone, before anytuning work begins. The VM, storage, and networking half of the bill is Terraform managed, and C3X prices it against the resource catalog in review.
FAQ
What is the difference between a job cluster and an all purpose cluster?
A job cluster is created for a single job run and terminated when it finishes, billing at the Jobs Compute DBU rate of roughly $0.15 on AWS Premium. An all purpose cluster is long lived, shared, and interactive, billing at roughly $0.55 per DBU. The pricing reflects that all purpose clusters sell continuous availability while job clusters sell one execution.
How much cheaper are job clusters?
About 73 percent cheaper on the DBU rate alone, since $0.15 versus $0.55 is a 3.7x difference. Including eliminated idle time the gap is usually larger. A 45 minute nightly pipeline might cost around $117 a month on a job cluster with on demand workers versus about $345 on an all purpose cluster with 60 minute auto termination.
Is job cluster startup time a problem?
Rarely. Startup is typically 2 to 6 minutes, which is irrelevant for scheduled batch work. If a job truly needs faster start, use a cluster pool, which keeps warm instances available at cloud provider cost with no DBU charge while idle, cutting startup to under a minute while still billing the run at the cheaper Jobs Compute rate.
Can multiple jobs share one job cluster?
Yes. Define them as multiple tasks within a single Databricks job and they share one job cluster for the whole run. That gives you one startup, the cheaper Jobs Compute rate, and automatic teardown at the end, which is strictly better than co locating separate scheduled jobs on a shared all purpose cluster.
What is the worst case cost of an all purpose cluster?
A cluster consuming 16 DBUs per hour left running continuously burns 11,520 DBUs a month, roughly $6,340 in DBUs plus about $2,020 in VM charges, over $8,000 for compute that may be doing nothing. This usually happens because auto termination was disabled during debugging and never turned back on.
What should I audit first?
List every scheduled job and note which cluster it attaches to. Any job pointing at an existing all purpose cluster should move to a job cluster with the same instance types, ideally with spot workers and an on demand driver. Then enforce auto termination at 30 minutes on remaining development clusters and remove the ability to disable it.
What to do next
Catch expensive platform configuration before it ships. C3X prices your Terraform resources in the pull request. Start with the quickstart.
Share this post
Try C3X on your own Terraform
Free and open source. No API key required. One command to install, one command to estimate.