Kubernetes CronJob and batch cost: the workloads that quietly keep nodes alive

Quick answer

Batch workloads are cheap in theory and expensive in practice because they are bursty and they block scale-down. A CronJob requesting 4 vCPU that runs 5 minutes every hour uses 6 GPU-free node-hours per month of actual compute but frequently keeps a $280 node alive continuously because the autoscaler cannot reclaim it between runs. The fixes are a dedicated spot node pool with scale-to-zero, low PriorityClass so batch preempts instead of provisioning, concurrencyPolicy Forbid, activeDeadlineSeconds, and ttlSecondsAfterFinished. Together these routinely take batch from 15 to 25 percent of cluster cost down to 3 to 6 percent.

Batch is the workload class that looks harmless on a cost review. Jobs start, do work, and exit. Nothing runs continuously, so nothing should cost continuously. Then you look at the node graph and see that the cluster never drops below 42 nodes, and the reason is a CronJob that runs for four minutes an hour.

Why finished jobs still cost money

Mechanism	Effect on cost
Job starts, autoscaler adds a node	Node provisioned in 2 to 4 minutes
Job finishes in 5 minutes	Node now under-utilized
scale-down-unneeded-time is 10 minutes	Node must idle 10 minutes to qualify
Next hourly run starts	Node used again, timer resets
Net effect	Node never removed, $280 per month

That is the core trap. A job that consumes 6 node-hours of real work per month holds a node for 730. The effective cost of that compute is 120 times what it should be. Completed pods left in the cluster make it worse, since Job pods persist after completion by default and count against the node until garbage collected.

The peak-sizing problem

Batch is also where peak sizing hurts most. A nightly ETL that needs 60 vCPU for 90 minutes forces the cluster to carry 60 vCPU of node capacity, and if scale-down is slow or blocked, it carries it all day. The honest cost of that job is 90 vCPU-hours, about $5. The cost actually paid is often 60 vCPU for 24 hours, about $78 per day.

Approach	Capacity held	Monthly cost
Shared pool, slow scale-down	60 vCPU always	$2,333
Shared pool, tuned scale-down	60 vCPU for 3 hours daily	$292
Dedicated spot pool, scale to zero	60 vCPU for 1.5 hours daily at 30 percent rate	$44

The configuration that makes batch cheap

Five settings do most of the work.

Setting	Value	Why
ttlSecondsAfterFinished	300	Removes completed pods so nodes empty
concurrencyPolicy	Forbid	Stops overlapping runs stacking capacity
activeDeadlineSeconds	Job-specific	Kills runaway jobs that pin nodes for hours
backoffLimit	2 to 4	Stops infinite retry loops burning capacity
successfulJobsHistoryLimit	1	Avoids accumulating completed Job objects

concurrencyPolicy is the one that causes incidents as well as cost. A CronJob scheduled every 5 minutes whose runs start taking 8 minutes will, with the default Allow policy, stack up indefinitely, each run requesting fresh capacity. Clusters have doubled in node count overnight from exactly this.

Give batch its own pool

The structural fix is a dedicated node pool: spot capacity, taints so only tolerating pods land there, and minSize 0 so it disappears when idle. Batch is the ideal spot workload because a reclaim means a retry, not an incident, and most batch frameworks retry by default.

Pair that with an aggressive scale-down configuration on that pool specifically. A batch pool can afford scale-down-unneeded-time of 2 minutes because there is no latency-sensitive traffic to protect. The shared production pool cannot.

Or give batch no capacity at all

The cheapest batch is batch that runs in the gaps. Assign a low or negative PriorityClass so batch pods are preemptible and schedule only where capacity already exists. In a cluster that carries 25 to 30 percent headroom for production burst, a large amount of batch work fits in that headroom at zero marginal node cost, and gets preempted automatically when production needs the space.

This works best for latency-insensitive, retry-safe, chunked work. It works poorly for a job with a hard deadline, which should get its own guaranteed capacity instead.

Spread the schedule

A final and very common waste: everything is scheduled at the top of the hour or at midnight. Twenty CronJobs at 0 0 * * * all provision capacity simultaneously, creating a peak that sizes the cluster for the rest of the day. Staggering them across a window turns one 60 vCPU peak into six 10 vCPU peaks, and the autoscaler handles that with a fraction of the capacity.

Add jitter to schedules, spread retries, and avoid aligning with the production traffic peak. Node pool definitions and taints live in Terraform, so price the batch pool against theresource catalog and compare it to what the same jobs cost on the shared pool. Pair it with spot nodes for the cheapest possible capacity and withidle cost analysis to confirm the pool really returns to zero.

FAQ

Why do Kubernetes CronJobs cost more than their runtime suggests?

Because they block scale-down. A job that runs five minutes every hour consumes about 6 node-hours of real work per month, but the autoscaler needs a node to be unneeded for a continuous period, typically 10 minutes, before removing it. The next run resets that timer, so the node is never reclaimed and costs a full $280 per month for compute worth a couple of dollars.

How do I stop Kubernetes batch jobs from holding nodes open?

Set ttlSecondsAfterFinished to about 300 so completed pods are removed and nodes can empty, set successfulJobsHistoryLimit to 1, set activeDeadlineSeconds so runaway jobs cannot pin nodes for hours, and cap backoffLimit at 2 to 4 to stop retry loops. Then move batch to a dedicated node pool with a short scale-down-unneeded-time and minSize 0.

What does concurrencyPolicy do for Kubernetes cost?

It controls whether a CronJob run can start while a previous run is still going. With the default Allow policy, a job scheduled every five minutes that starts taking eight minutes stacks up indefinitely, each run requesting fresh capacity. Clusters have doubled in node count overnight from this. Setting concurrencyPolicy to Forbid prevents it.

Should Kubernetes batch workloads run on spot nodes?

Yes, batch is the ideal spot workload because a reclaim means a retry rather than an incident, and most batch frameworks retry by default. A dedicated spot pool with taints and minSize 0 takes a nightly 60 vCPU ETL from roughly $2,333 per month on a slow-scaling shared pool to about $44 per month.

Can Kubernetes batch jobs run at zero marginal cost?

Often, yes. Assign a low or negative PriorityClass so batch pods are preemptible and schedule only into existing headroom. A cluster carrying 25 to 30 percent spare capacity for production burst can absorb a lot of retry-safe, latency-insensitive batch work at no additional node cost, with automatic preemption when production needs the space.

How does C3X help with Kubernetes batch cost?

Batch node pools, taints, and capacity types are declared in Terraform, so C3X prices them against a live catalog. That lets you compare what a dedicated scale-to-zero spot pool costs against leaving the same jobs on the shared production pool, as a concrete monthly figure in the pull request rather than an argument about autoscaler behaviour.

What to do next

Price the batch pool against the shared one. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.