The right spot to on-demand node mix for Kubernetes: how far can you push it
Spot nodes cost 60 to 90 percent less but can disappear with two minutes of warning. The question is not whether to use them but what percentage of the cluster they should be. Here is how to choose the ratio by workload.
Quick answer
Spot capacity runs 60 to 90 percent below on-demand, so the mix drives most of a cluster's node cost. A practical split is an on-demand baseline sized for the traffic you must serve during a full spot reclaim, typically 25 to 40 percent of capacity, with the remaining 60 to 75 percent on spot. At a 70 percent spot ratio and a 70 percent spot discount, a $20,000 node bill becomes $10,200, a 49 percent saving. Non-production and batch can push to 90 to 100 percent spot. The safety requirements are instance type diversification across at least six types, PodDisruptionBudgets, topology spread across both pools, and a node termination handler.
Spot instances are the largest single discount available in Kubernetes, and unlike right-sizing they need no application change. The engineering question is the ratio: how much of the cluster can run on capacity that might vanish, and how do you size the part that cannot.
What the discount is worth
| Spot share of capacity | Blended rate vs on-demand | On a $20,000 node bill |
|---|---|---|
| 0 percent | 100 percent | $20,000 |
| 30 percent | 79 percent | $15,800 |
| 50 percent | 65 percent | $13,000 |
| 70 percent | 51 percent | $10,200 |
| 90 percent | 37 percent | $7,400 |
Assuming a 70 percent spot discount, which is typical for general-purpose families in large regions. The returns are close to linear, so every 10 points of spot share is worth about 7 percent of the node bill. Moving from 30 percent to 70 percent spot on a $20,000 bill is $5,600 per month.
Sizing the on-demand baseline
The baseline is not a comfort number, it is an answer to a question: if every spot node in the cluster went away in the same five minutes, what must still serve traffic? For most services that is the capacity needed for p50 traffic with degraded but acceptable latency, not p99 with full headroom.
| Workload class | On-demand baseline | Spot share |
|---|---|---|
| Production user-facing API | 40 percent | 60 percent |
| Internal services | 25 percent | 75 percent |
| Async workers and queues | 10 percent | 90 percent |
| Batch and CI | 0 percent | 100 percent |
| Staging and dev | 0 to 10 percent | 90 to 100 percent |
| Stateful databases in cluster | 100 percent | 0 percent |
A cluster running all of those in proportion typically lands between 60 and 75 percent spot overall, which is the sweet spot where savings are large and the on-demand floor still covers a full reclaim.
Diversification is the real safety mechanism
The risk that matters is not one node being reclaimed, it is a correlated reclaim of every node of the same instance type in the same zone. A node pool restricted to one instance type in one zone can lose all its capacity at once. A pool spanning six instance types across three zones almost never does, because those spot pools drain independently.
Practically: allow at least 6 to 10 instance types of similar shape, m5, m5a, m5n, m6i, m6a, m7i for a general-purpose pool, across all zones in the region. Modern provisioners choose per pending pod, which makes this easy. Capacity-optimized allocation strategies pick the deepest pools, which materially lowers interruption rates compared with lowest-price strategies.
Making workloads survive reclaim
| Control | What it prevents |
|---|---|
| Node termination handler | Hard kills, by draining on the 2-minute warning |
| PodDisruptionBudget | All replicas of a service leaving at once |
| topologySpreadConstraints | All replicas landing on one instance type or zone |
| Node affinity weighting | Critical pods preferring on-demand nodes |
| terminationGracePeriodSeconds | In-flight requests being cut off |
| Checkpointing for long jobs | Losing hours of batch work |
The pattern for splitting a single Deployment across both pools is a required topology spread across a node label such as capacity type, plus a preferred node affinity that weights on-demand for the first replicas. That way a 10-replica service keeps 3 or 4 replicas on on-demand nodes and lets the rest ride spot.
Where the mix goes wrong
Three failure modes recur. First, putting stateful workloads on spot: a database pod evicted with a 2 minute warning may not finish a clean shutdown, and the replacement has to re-sync. Keep stateful sets on on-demand unless the storage layer genuinely tolerates node loss.
Second, over-narrow instance selection, which recreates correlated risk. Third, forgetting that spot prices move: the discount is usually 60 to 90 percent but tightens for popular families during regional demand spikes. Diversification protects against price as well as availability.
Combining with commitments
Spot and commitments are complements, not alternatives. Cover the on-demand baseline with a Savings Plan or committed use discount, since that capacity runs continuously, and leave the variable spot layer uncommitted. A cluster with 30 percent on-demand baseline covered at a 30 percent commitment discount and 70 percent spot at a 70 percent discount runs at about 42 percent of full on-demand cost. See the spot nodes guide for implementation detail and the combined strategy for the commitment side.
Capacity types and pool ratios are declared in Terraform. Price the mix against the resource catalog so a proposed change from 40 percent to 70 percent spot carries its monthly saving in the pull request, and so the on-demand baseline stays a deliberate number rather than whatever was there last year.
FAQ
What percentage of a Kubernetes cluster should run on spot?
Most production clusters land between 60 and 75 percent spot. Size the on-demand baseline to answer one question: if every spot node vanished in five minutes, what must still serve traffic. That is usually p50 traffic with degraded latency, around 25 to 40 percent of capacity. Batch, CI, and non-production can run at 90 to 100 percent spot.
How much does a spot and on-demand mix save in Kubernetes?
Assuming a typical 70 percent spot discount, every 10 points of spot share saves about 7 percent of the node bill. A $20,000 node bill costs $15,800 at 30 percent spot, $13,000 at 50 percent, $10,200 at 70 percent, and $7,400 at 90 percent. Moving from 30 to 70 percent spot is worth about $5,600 per month on that bill.
How do I make spot nodes safe in Kubernetes?
Diversify across at least six to ten instance types of similar shape and all zones in the region so reclaims are uncorrelated, use a capacity-optimized allocation strategy, run a node termination handler that drains on the two-minute warning, define PodDisruptionBudgets, spread replicas with topologySpreadConstraints across the capacity-type label, and set a realistic terminationGracePeriodSeconds.
Can one Deployment span spot and on-demand nodes?
Yes, and it is the standard pattern. Use a required topologySpreadConstraint across a node label such as capacity type, plus a preferred node affinity weighting on-demand nodes. A 10-replica service then keeps three or four replicas on on-demand capacity as a floor while the remaining replicas ride spot at a fraction of the price.
Should stateful workloads run on spot nodes?
Generally no. A database pod reclaimed with two minutes of warning may not complete a clean shutdown, and its replacement has to re-sync, which costs time and cross-zone transfer. Keep StatefulSets on on-demand capacity unless the storage layer genuinely tolerates abrupt node loss and re-replication is cheap and fast.
How does C3X help choose a spot to on-demand ratio?
Capacity types and node pool ratios are declared in Terraform, so C3X prices them against a live catalog before merge. A proposed shift from 40 percent to 70 percent spot shows its monthly saving in the pull request, and the on-demand baseline stays a deliberate, priced number rather than whatever the node group happened to be set to last year.
What to do next
Price the spot mix before you change it. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.
Related posts
- Kubernetes spot nodes guide: cutting node cost with interruptible capacity
Spot nodes can cut Kubernetes compute cost by well over half, but they can be reclaimed at short notice, so they suit fault-tolerant workloads with graceful handling. Here is how to use spot safely and how much it saves.
- Kubernetes GPU node pool cost: why one idle A100 costs more than 30 web servers
GPU nodes cost 10 to 40 times a general-purpose node per hour, and Kubernetes gives you a whole GPU per pod by default. Idle GPU nodes are the most expensive waste in any cluster. Here are the rates and the sharing strategies.
- Kubernetes node pool sizing for cost: big nodes, small nodes, and the waste between
Node size is one of the highest-leverage cost decisions in Kubernetes and most teams pick by habit. Too small and system overhead eats your capacity, too big and a single pod strands half a node. Here is how to size node pools with real numbers.
- Kubernetes cost per pod explained: from node price to pod price
A pod has no price tag; it inherits a slice of its node's cost based on the CPU and memory it requests or uses. Understanding that translation reveals why over-requesting is expensive and where pod-level waste hides. Here is the model.
Share this post
Try C3X on your own Terraform
Free and open source. No API key required. One command to install, one command to estimate.