H100 vs A100 cost per training hour: which GPU is actually cheaper

Quick answer

On AWS on-demand in us-east-1, a p5.48xlarge (8x H100 80GB) lists around $98.32 per hour, about $12.29 per GPU-hour, while a p4d.24xlarge (8x A100 40GB) lists around $32.77 per hour, about $4.10 per GPU-hour. The H100 is roughly 3x the price per hour. It is cheaper overall only if it finishes the same training run at least 3x faster. For large transformer training in mixed precision, H100 throughput is commonly 2x to 4x an A100, so H100 wins on dense, well-optimized, large-batch jobs and loses on small models, data-bound pipelines, and anything that leaves the GPU idle. Always compare dollars per epoch or dollars per token, not dollars per hour.

The question is never which GPU has the lower hourly rate. It is which GPU finishes the run for less money. An H100 instance costs around three times an A100 instance per GPU-hour, so the H100 only saves money if it is more than three times more productive on your specific workload. Sometimes it is. Often it is not, and the difference between those two cases is worth tens of thousands of dollars on a single training campaign.

The list prices

Instance	GPUs	On-demand per hour	Per GPU-hour
p5.48xlarge (AWS)	8x H100 80GB	about $98.32	about $12.29
p4de.24xlarge (AWS)	8x A100 80GB	about $40.96	about $5.12
p4d.24xlarge (AWS)	8x A100 40GB	about $32.77	about $4.10
a3-highgpu-8g (GCP)	8x H100 80GB	about $88	about $11.00
a2-highgpu-1g (GCP)	1x A100 40GB	about $3.67	about $3.67
ND96isr H100 v5 (Azure)	8x H100 80GB	about $98	about $12.25

Prices are US list rates and move with region and commitment. The shape holds everywhere: H100 capacity sits at roughly 2.5x to 3x the A100 rate per GPU-hour. A broader view of the families is in the GPU instance cost comparison.

The only metric that matters: cost per unit of work

Take a concrete run. Suppose fine-tuning a 7B parameter model on 20 billion tokens takes 60 GPU-hours on A100 80GB hardware. At $5.12 per GPU-hour that is about $307. If the same run takes 22 GPU-hours on H100 hardware because of faster tensor cores, FP8 support, and higher memory bandwidth, then at $12.29 per GPU-hour it costs about $270. The H100 wins, but only by 12 percent, not by the 3x headline the speedup suggests. If the speedup is only 2x, the H100 run costs about $369 and the A100 is 17 percent cheaper.

That is the whole decision in one table:

H100 speedup vs A100	Relative cost of H100 run	Verdict
1.5x	1.6x the A100 cost	A100 clearly cheaper
2.0x	1.2x the A100 cost	A100 cheaper
2.4x	about break-even	Tie, pick on wall-clock
3.0x	0.8x the A100 cost	H100 cheaper
4.0x	0.6x the A100 cost	H100 clearly cheaper

When the H100 actually delivers 3x or more

The big speedups come from dense matrix work at large batch sizes with FP8 or BF16 precision, models that fit comfortably in 80GB so you are not fighting memory, and pipelines that keep the GPU fed. Transformer pretraining with a well-tuned data loader is the canonical case. H100 memory bandwidth of about 3.35 TB/s against roughly 2 TB/s on A100 80GB also helps memory-bound attention kernels.

The small speedups come from everything else. Small models where kernel launch overhead dominates, jobs bottlenecked on data loading from object storage, workloads in FP32 that never touch the newer tensor core paths, and any run where a single-node code path was written for A100 and never retuned. If your utilization is 40 percent, buying a faster GPU mostly buys faster idling, which is the trap covered in GPU instance cost optimization.

Do not forget the rest of the bill

A p5.48xlarge is not just GPUs. It brings 192 vCPUs, 2 TB of RAM, and 30 TB of local NVMe, all included in that hourly rate, plus 3200 Gbps of network fabric that makes multi-node scaling viable. If your job needs that interconnect to scale past one node, the comparison changes again: an eight-node A100 cluster that scales at 70 percent efficiency can lose to a four-node H100 cluster that scales at 90 percent. Storage and egress ride on top of both, and checkpoint traffic at $0.023 per GB-month on S3 Standard is small but cross-region copies at $0.02 per GB are not.

Commitment and spot change the ranking

Spot capacity for A100 classes is usually easier to obtain than for H100, and discounts of 50 to 70 percent are common on both when available. An A100 at $1.60 spot beats an H100 at $12.29 on-demand for almost any speedup ratio, provided your training is checkpointed and restartable. That is the argument in spot versus on-demand, applied to GPUs. Reserved or committed-use pricing cuts 30 to 60 percent off both, but it also locks you into a generation, which is a real risk when hardware turns over every 18 to 24 months.

How to decide without guessing

Benchmark one epoch, or even a few hundred steps, on both. Record samples per second, then divide the hourly rate by it to get dollars per million samples or dollars per million tokens. That single number ends the argument. Do this before committing a multi-week training budget, because a 20 percent error on a run that costs $80,000 is $16,000. Model the infrastructure in Terraform and price it against the resource catalog so the GPU choice is costed at design time rather than discovered at the end of the month, the same pre-deploy discipline described in ML training cost optimization.

FAQ

Is an H100 cheaper than an A100 for training?

Only if it is more than about 2.4x faster on your workload, because H100 instances cost roughly 3x the A100 rate per GPU-hour on AWS list pricing (about $12.29 versus about $4.10 to $5.12). On large transformer training with FP8 or BF16 and big batches, H100 speedups of 2x to 4x are common, so it often wins. On small models, data-bound pipelines, or FP32 code, the A100 is usually cheaper per unit of work.

What does an H100 cost per hour on AWS?

A p5.48xlarge with 8 H100 80GB GPUs lists at roughly $98.32 per hour on-demand in us-east-1, which works out to about $12.29 per GPU-hour. That rate includes 192 vCPUs, about 2 TB of RAM, 30 TB of local NVMe, and high-bandwidth networking. GCP a3-highgpu-8g and Azure ND96isr H100 v5 sit in a similar range, around $88 to $98 per hour for eight GPUs.

What does an A100 cost per hour?

On AWS, p4d.24xlarge with 8 A100 40GB GPUs lists around $32.77 per hour, about $4.10 per GPU-hour, and p4de.24xlarge with 80GB GPUs lists around $40.96 per hour, about $5.12 per GPU-hour. On GCP, a single A100 40GB via a2-highgpu-1g runs about $3.67 per hour. Spot and committed-use pricing can cut these by 30 to 70 percent.

How do I compare GPU cost properly?

Compare dollars per unit of work, not dollars per hour. Benchmark a few hundred training steps on each GPU type, record samples or tokens per second, then divide the hourly rate by throughput to get cost per million samples or per million tokens. That single number accounts for both price and speed and settles the comparison for your specific model, batch size, and precision.

Does spot pricing change the H100 versus A100 decision?

Significantly. A100 spot capacity is generally easier to obtain than H100 spot, and discounts of 50 to 70 percent are common. An A100 at spot rates beats an H100 at on-demand rates for nearly any realistic speedup ratio, provided your training checkpoints frequently and restarts cleanly. Interruption tolerance is what makes that saving accessible.

How does C3X help with GPU cost decisions?

C3X prices GPU instances defined in Terraform against a live catalog before you deploy, so a change from an A100 instance family to an H100 family shows its hourly and monthly cost delta in the pull request. Combined with your own throughput benchmarks, that turns the GPU choice into a cost-per-unit-of-work decision made at design time rather than a surprise on the invoice.

What to do next

Price your GPU training cluster before you launch it. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.