GPU cost per teraflop compared: normalizing hourly rates into value

Quick answer

Normalizing cloud GPU prices by delivered compute changes the ranking completely. Using dense FP16 tensor throughput and list on-demand rates, an L4 at about $0.805 per hour for roughly 242 TFLOPS costs about $0.0033 per TFLOPS-hour, an L40S at about $1.861 for roughly 362 TFLOPS costs about $0.0051, an A100 80GB at about $5.12 for roughly 312 TFLOPS costs about $0.0164, and an H100 at about $12.29 for roughly 990 TFLOPS costs about $0.0124. Cheap inference GPUs win on raw FLOPS per dollar, while big GPUs win on memory capacity and bandwidth per dollar, which is what large models actually need.

A cloud GPU price list is nearly useless as a decision tool, because the prices span fifteen to one and so does the hardware. The fix is normalization: divide the hourly rate by the capability you actually consume. Do that twice, once for compute and once for memory bandwidth, and the right choice for a given workload becomes obvious.

The normalized table

GPU	Per GPU-hour	Dense FP16 TFLOPS	$ per TFLOPS-hour	Memory
T4	about $0.53	about 65	about $0.0081	16 GB
L4	about $0.805	about 242	about $0.0033	24 GB
A10G	about $1.01	about 125	about $0.0080	24 GB
L40S	about $1.861	about 362	about $0.0051	48 GB
A100 80GB	about $5.12	about 312	about $0.0164	80 GB
H100 80GB	about $12.29	about 990	about $0.0124	80 GB

TFLOPS figures are vendor dense tensor numbers for FP16 or BF16 without sparsity, and real achieved throughput is typically 30 to 60 percent of them. Prices are AWS-equivalent US on-demand list rates, derived from the instance families discussed in GPU instance pricing explained.

What this ranking says

The L4 is the best raw compute value in the list by a wide margin, roughly five times better per TFLOPS-hour than an A100. The H100 is better value per TFLOPS than the A100 despite costing 2.4 times more per hour, which is the central point of the current generation: it is not just faster, it is cheaper per unit of compute. And the T4, still the default choice in countless deployments, is now poor value on this metric, beaten by the L4 on both price-performance and memory.

Why the cheap GPUs do not always win

GPU	Bandwidth	$ per GB/s-hour	$ per GB of memory-hour
T4	about 320 GB/s	about $0.00166	about $0.033
L4	about 300 GB/s	about $0.00268	about $0.034
L40S	about 864 GB/s	about $0.00215	about $0.039
A100 80GB	about 2,039 GB/s	about $0.00251	about $0.064
H100 80GB	about 3,350 GB/s	about $0.00367	about $0.154

Token generation in a transformer is memory-bandwidth-bound, not compute-bound, because each generated token reads the entire model from memory. For that workload, dollars per GB/s-hour is the metric that matters, and the ranking tightens considerably: the T4 looks good here, the L4 looks mediocre, and the H100 premium is real but not five-fold.

The constraint that overrides both metrics

Memory capacity is binary. A model either fits or it does not. A 70B model in FP16 needs about 140 GB of weights, so it requires at least two 80 GB GPUs regardless of how attractive the L4's price-performance looks, because you cannot split a model across six L4s without an interconnect that makes it pointless. Any price-performance analysis is therefore filtered first by capacity: eliminate the GPUs that cannot hold your model plus KV cache, then rank what remains.

Applying it to three workloads

Workload	Bound by	Metric to use	Typical winner
Embedding generation	Compute	$ per TFLOPS-hour	L4
Image generation	Compute and memory	Both	L40S
LLM token generation	Bandwidth and capacity	$ per GB/s-hour	L40S or H100
Large model training	Compute and interconnect	$ per TFLOPS-hour	H100
Small model batch scoring	Compute	$ per TFLOPS-hour	L4

What the instance bundles changes the math again

You never buy a bare GPU in the cloud. A p5.48xlarge includes 192 vCPUs, about 2 TB of RAM, 30 TB of local NVMe, and 3,200 Gbps of networking in its roughly $98.32 per hour. A g6.xlarge includes 4 vCPUs, 16 GB of RAM, and 250 GB of NVMe in its roughly $0.805. If your workload needs heavy CPU-side preprocessing or a large in-memory dataset, the big instance is not as overpriced as its per-teraflop number suggests, because the alternative is renting that CPU and memory separately and paying to move data between them. Conversely, if your pipeline is pure GPU work, every bundled vCPU you do not use is waste embedded in the hourly rate, which is an argument for the single-GPU shapes where the bundle is smallest relative to the accelerator.

Use it as a screen, not a verdict

Normalized metrics narrow six options to two. The final choice still needs a benchmark on your actual model, because achieved utilization varies by 2x between well-tuned and naive implementations on the same hardware. But screening first saves you from benchmarking everything, and it prevents the common error of picking on hourly rate alone. Once you have the shortlist, define the instances in Terraform and price them against the resource catalog so the fleet-level monthly cost of each candidate is on the table alongside the per-hour rate.

FAQ

Which cloud GPU has the best cost per teraflop?

Among common cloud GPUs, the L4 leads at roughly $0.0033 per TFLOPS-hour, based on about $0.805 per hour for roughly 242 dense FP16 TFLOPS. The L40S follows at about $0.0051, the H100 at about $0.0124, and the A100 80GB at about $0.0164. Notably the H100 is better value per unit of compute than the A100 despite costing about 2.4 times more per hour.

Why is cost per teraflop not the whole story?

Because many AI workloads are bound by memory bandwidth or capacity rather than compute. Transformer token generation reads the entire model from memory for each token, making it bandwidth-bound, so dollars per GB/s-hour is the relevant metric there. And memory capacity is binary: a 70B model in FP16 needs about 140 GB of weights, so it simply cannot run on a 24 GB GPU regardless of price-performance.

How do I pick a GPU for token generation?

Filter first by memory capacity, since the model weights plus KV cache must fit, then rank the survivors by dollars per GB/s-hour rather than per teraflop. On that measure the T4 is about $0.00166, the L40S about $0.00215, the A100 80GB about $0.00251, and the H100 about $0.00367. The H100 premium is real but far smaller on bandwidth than on hourly rate.

Is the T4 still good value?

Not on compute. At about $0.53 per hour for roughly 65 dense FP16 TFLOPS, it costs about $0.0081 per TFLOPS-hour, roughly 2.5 times worse than the L4. It still looks reasonable on dollars per GB/s-hour because its bandwidth is comparable to the L4's, and its low absolute hourly rate wins when a GPU sits mostly idle, but for saturated compute-bound work it has been superseded.

Do published TFLOPS numbers reflect real performance?

No. Vendor dense tensor figures assume ideal kernels and full occupancy, and real achieved throughput is typically 30 to 60 percent of the headline number, varying by as much as 2x between a well-tuned and a naive implementation on identical hardware. Use normalized metrics to narrow six candidates to two, then benchmark those two on your actual model before committing.

How does C3X help compare GPU options?

C3X prices the instance families in your Terraform against a live catalog, so once normalized metrics narrow the shortlist, the actual fleet-level monthly cost of each candidate is visible in the pull request. That connects the per-teraflop screening to the number that appears on the invoice, including the supporting CPU, memory, and storage bundled into each GPU instance type.

What to do next

Turn GPU price lists into decisions. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.