The right spot to on-demand node mix for Kubernetes: how far can you push it

Quick answer

Spot capacity runs 60 to 90 percent below on-demand, so the mix drives most of a cluster's node cost. A practical split is an on-demand baseline sized for the traffic you must serve during a full spot reclaim, typically 25 to 40 percent of capacity, with the remaining 60 to 75 percent on spot. At a 70 percent spot ratio and a 70 percent spot discount, a $20,000 node bill becomes $10,200, a 49 percent saving. Non-production and batch can push to 90 to 100 percent spot. The safety requirements are instance type diversification across at least six types, PodDisruptionBudgets, topology spread across both pools, and a node termination handler.

Spot instances are the largest single discount available in Kubernetes, and unlike right-sizing they need no application change. The engineering question is the ratio: how much of the cluster can run on capacity that might vanish, and how do you size the part that cannot.

What the discount is worth

Spot share of capacity	Blended rate vs on-demand	On a $20,000 node bill
0 percent	100 percent	$20,000
30 percent	79 percent	$15,800
50 percent	65 percent	$13,000
70 percent	51 percent	$10,200
90 percent	37 percent	$7,400

Assuming a 70 percent spot discount, which is typical for general-purpose families in large regions. The returns are close to linear, so every 10 points of spot share is worth about 7 percent of the node bill. Moving from 30 percent to 70 percent spot on a $20,000 bill is $5,600 per month.

Sizing the on-demand baseline

The baseline is not a comfort number, it is an answer to a question: if every spot node in the cluster went away in the same five minutes, what must still serve traffic? For most services that is the capacity needed for p50 traffic with degraded but acceptable latency, not p99 with full headroom.

Workload class	On-demand baseline	Spot share
Production user-facing API	40 percent	60 percent
Internal services	25 percent	75 percent
Async workers and queues	10 percent	90 percent
Batch and CI	0 percent	100 percent
Staging and dev	0 to 10 percent	90 to 100 percent
Stateful databases in cluster	100 percent	0 percent

A cluster running all of those in proportion typically lands between 60 and 75 percent spot overall, which is the sweet spot where savings are large and the on-demand floor still covers a full reclaim.

Diversification is the real safety mechanism

The risk that matters is not one node being reclaimed, it is a correlated reclaim of every node of the same instance type in the same zone. A node pool restricted to one instance type in one zone can lose all its capacity at once. A pool spanning six instance types across three zones almost never does, because those spot pools drain independently.

Practically: allow at least 6 to 10 instance types of similar shape, m5, m5a, m5n, m6i, m6a, m7i for a general-purpose pool, across all zones in the region. Modern provisioners choose per pending pod, which makes this easy. Capacity-optimized allocation strategies pick the deepest pools, which materially lowers interruption rates compared with lowest-price strategies.

Making workloads survive reclaim

Control	What it prevents
Node termination handler	Hard kills, by draining on the 2-minute warning
PodDisruptionBudget	All replicas of a service leaving at once
topologySpreadConstraints	All replicas landing on one instance type or zone
Node affinity weighting	Critical pods preferring on-demand nodes
terminationGracePeriodSeconds	In-flight requests being cut off
Checkpointing for long jobs	Losing hours of batch work

The pattern for splitting a single Deployment across both pools is a required topology spread across a node label such as capacity type, plus a preferred node affinity that weights on-demand for the first replicas. That way a 10-replica service keeps 3 or 4 replicas on on-demand nodes and lets the rest ride spot.

Where the mix goes wrong

Three failure modes recur. First, putting stateful workloads on spot: a database pod evicted with a 2 minute warning may not finish a clean shutdown, and the replacement has to re-sync. Keep stateful sets on on-demand unless the storage layer genuinely tolerates node loss.

Second, over-narrow instance selection, which recreates correlated risk. Third, forgetting that spot prices move: the discount is usually 60 to 90 percent but tightens for popular families during regional demand spikes. Diversification protects against price as well as availability.

Combining with commitments

Spot and commitments are complements, not alternatives. Cover the on-demand baseline with a Savings Plan or committed use discount, since that capacity runs continuously, and leave the variable spot layer uncommitted. A cluster with 30 percent on-demand baseline covered at a 30 percent commitment discount and 70 percent spot at a 70 percent discount runs at about 42 percent of full on-demand cost. See thespot nodes guide for implementation detail andthe combined strategy for the commitment side.

Capacity types and pool ratios are declared in Terraform. Price the mix against theresource catalog so a proposed change from 40 percent to 70 percent spot carries its monthly saving in the pull request, and so the on-demand baseline stays a deliberate number rather than whatever was there last year.

FAQ

What percentage of a Kubernetes cluster should run on spot?

Most production clusters land between 60 and 75 percent spot. Size the on-demand baseline to answer one question: if every spot node vanished in five minutes, what must still serve traffic. That is usually p50 traffic with degraded latency, around 25 to 40 percent of capacity. Batch, CI, and non-production can run at 90 to 100 percent spot.

How much does a spot and on-demand mix save in Kubernetes?

Assuming a typical 70 percent spot discount, every 10 points of spot share saves about 7 percent of the node bill. A $20,000 node bill costs $15,800 at 30 percent spot, $13,000 at 50 percent, $10,200 at 70 percent, and $7,400 at 90 percent. Moving from 30 to 70 percent spot is worth about $5,600 per month on that bill.

How do I make spot nodes safe in Kubernetes?

Diversify across at least six to ten instance types of similar shape and all zones in the region so reclaims are uncorrelated, use a capacity-optimized allocation strategy, run a node termination handler that drains on the two-minute warning, define PodDisruptionBudgets, spread replicas with topologySpreadConstraints across the capacity-type label, and set a realistic terminationGracePeriodSeconds.

Can one Deployment span spot and on-demand nodes?

Yes, and it is the standard pattern. Use a required topologySpreadConstraint across a node label such as capacity type, plus a preferred node affinity weighting on-demand nodes. A 10-replica service then keeps three or four replicas on on-demand capacity as a floor while the remaining replicas ride spot at a fraction of the price.

Should stateful workloads run on spot nodes?

Generally no. A database pod reclaimed with two minutes of warning may not complete a clean shutdown, and its replacement has to re-sync, which costs time and cross-zone transfer. Keep StatefulSets on on-demand capacity unless the storage layer genuinely tolerates abrupt node loss and re-replication is cheap and fast.

How does C3X help choose a spot to on-demand ratio?

Capacity types and node pool ratios are declared in Terraform, so C3X prices them against a live catalog before merge. A proposed shift from 40 percent to 70 percent spot shows its monthly saving in the pull request, and the on-demand baseline stays a deliberate, priced number rather than whatever the node group happened to be set to last year.

What to do next

Price the spot mix before you change it. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.