Kubernetes GPU node pool cost: why one idle A100 costs more than 30 web servers

Quick answer

GPU nodes dominate any cluster bill that has them. On AWS, g4dn.xlarge with a T4 is $0.526 per hour or $384 per month, g5.xlarge with an A10G is $1.006 or $734, p3.2xlarge with a V100 is $3.06 or $2,234, and p4d.24xlarge with eight A100s is $32.77 or $23,922. GCP a2-highgpu-1g with one A100 is about $3.67 per hour. Kubernetes allocates whole GPUs per pod by default, so a pod using 8 percent of an A100 still costs the full $2,679 per month. Time-slicing, MIG partitioning, spot GPU pools, and scale-to-zero node groups are where the savings are, commonly 50 to 80 percent.

Adding a GPU node pool changes the shape of a Kubernetes bill completely. A cluster that ran on $280 per month general-purpose nodes suddenly has line items where a single instance costs more than the entire previous cluster. And because the Kubernetes device plugin model hands out whole GPUs, utilization is usually terrible.

What GPU nodes cost

Instance	GPU	Per hour	Per month 24x7
AWS g4dn.xlarge	1x T4 16 GB	$0.526	$384
AWS g5.xlarge	1x A10G 24 GB	$1.006	$734
AWS g5.12xlarge	4x A10G	$5.672	$4,140
AWS p3.2xlarge	1x V100 16 GB	$3.06	$2,234
AWS p4d.24xlarge	8x A100 40 GB	$32.77	$23,922
GCP a2-highgpu-1g	1x A100 40 GB	about $3.67	$2,679
Azure NC4as T4 v3	1x T4	about $0.526	$384
Azure NC6s v3	1x V100	about $3.06	$2,234

For scale: one p4d.24xlarge running continuously costs about as much as 85 m5.2xlarge nodes. A single idle A100 instance wastes more per month than most teams' entire non-production environment.

The whole-GPU allocation problem

The NVIDIA device plugin exposes nvidia.com/gpu as an integer resource. A pod requests 1 and gets an entire physical GPU, exclusively, for its whole lifetime. That is correct for a training job saturating the card. It is wildly wasteful for inference serving a small model, for a notebook a data scientist has open while in meetings, or for a CI job that runs tests for four minutes.

Workload	Typical GPU utilization	Monthly cost on A10G	Effective value of work
Training job, saturated	85 to 95 percent	$734	$734
Small model inference	8 to 20 percent	$734	$59 to $147
Notebook, interactive	3 to 10 percent	$734	$22 to $73
CI test suite	under 2 percent	$734	under $15

Sharing strategies

Time-slicing is the simplest. The NVIDIA device plugin can advertise one physical GPU as several logical replicas, so multiple pods share it by interleaving. There is no memory isolation and no performance guarantee, so it fits notebooks, light inference, and CI, not production training. Advertising a T4 as 4 replicas takes the effective cost per workload from $384 to $96 per month.

MIG, Multi-Instance GPU, is the hardware-backed version available on A100 and H100. It partitions a card into up to seven isolated instances with dedicated memory and compute slices. An A100 40 GB split into seven 5 GB instances serves seven inference workloads with real isolation, taking effective cost from $2,679 to $383 per workload per month. MIG profiles are configured at the node level, so run a MIG-enabled pool for inference and a whole-GPU pool for training.

Scale to zero is mandatory

A general-purpose node pool that never scales to zero wastes $280 per month. A GPU pool that never scales to zero wastes $2,234. The arithmetic makes scale-to-zero non-optional for GPU pools: set minSize to 0, taint the nodes so only GPU workloads land there, and make sure no DaemonSet without a matching toleration pins the node open. That last point catches many teams, since a logging or monitoring DaemonSet tolerating all taints will keep a $3 per hour node alive forever.

Batch and training workloads should be queued, not resident. Submit as Jobs, let the node pool scale up, run, and scale back to zero. A team running 60 hours of training per month on demand pays $60 on a g5 pool that scales to zero, versus $734 for a permanently running node, a 92 percent reduction.

Spot GPUs

GPU spot discounts are often larger than general-purpose ones, commonly 60 to 70 percent, because demand is lumpy. A g5.xlarge at $1.006 on demand frequently sits near $0.35 on spot. For checkpointed training, this is the single biggest lever available: a 200-hour training run costs $201 on demand and roughly $70 on spot. The requirement is checkpointing every 10 to 20 minutes and a job controller that resumes from the last checkpoint. Our spot nodes guide covers the interruption handling.

Watch the attached costs too

GPU instances come with large local NVMe and often large attached block volumes for datasets. A p4d with 8 TB of gp3 attached adds $640 per month. Dataset transfer between regions at $0.02 per GB adds up fast when training pulls a 2 TB dataset repeatedly. Keep data in the same region and zone as the GPU pool.

GPU node pools are defined in Terraform, which means their cost is knowable before the pool exists. Price the instance type and pool size against the resource catalog so a change from g4dn to p4d shows as a delta of roughly $23,000 per month in the pull request, where it belongs.

FAQ

How much do GPU nodes cost in Kubernetes?

On AWS, g4dn.xlarge with a T4 is $0.526 per hour or $384 per month, g5.xlarge with an A10G is $1.006 or $734, p3.2xlarge with a V100 is $3.06 or $2,234, and p4d.24xlarge with eight A100s is $32.77 or $23,922. GCP a2-highgpu-1g with one A100 is about $3.67 per hour. One p4d running continuously costs roughly as much as 85 general-purpose m5.2xlarge nodes.

Can multiple pods share a GPU in Kubernetes?

Yes, two ways. Time-slicing lets the NVIDIA device plugin advertise one physical GPU as several logical replicas that interleave, with no memory isolation, suitable for notebooks, light inference, and CI. MIG partitions A100 and H100 cards into up to seven hardware-isolated instances with dedicated memory. Splitting an A100 seven ways takes effective cost per workload from $2,679 to $383 per month.

Why is my GPU node pool so expensive when utilization is low?

Because the NVIDIA device plugin allocates whole GPUs. A pod requesting one unit of nvidia.com/gpu gets an entire physical card exclusively for its lifetime, so an inference workload using 8 to 20 percent of an A10G still costs the full $734 per month. Time-slicing, MIG partitioning, and scale-to-zero node pools are the ways to close that gap.

Should GPU node pools scale to zero?

Yes, it is effectively mandatory. An idle general-purpose node wastes $280 per month while an idle V100 node wastes $2,234. Set minSize to 0, taint GPU nodes so only GPU workloads schedule there, and check that no DaemonSet tolerating all taints pins the node open. A team running 60 training hours per month pays about $60 on a scale-to-zero pool versus $734 for a resident node.

Are spot GPU instances worth it for Kubernetes?

For checkpointed training, yes, and the discounts are often larger than general-purpose spot at 60 to 70 percent. A g5.xlarge at $1.006 on demand frequently runs near $0.35 on spot, so a 200-hour training run costs $201 on demand versus roughly $70 on spot. The requirement is checkpointing every 10 to 20 minutes and a job controller that resumes from the last checkpoint.

How does C3X help with Kubernetes GPU cost?

GPU node pools are defined in Terraform, so C3X prices the instance type and pool size against a live catalog before the pool exists. A change from a g4dn pool to a p4d pool shows as a delta of roughly $23,000 per month in the pull request, which is the only moment where questioning the instance choice is cheap.

What to do next

Price GPU pools before they run for a month. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.