L4 vs T4 GPU inference cost: the quiet upgrade that pays for itself

Quick answer

On AWS, g4dn.xlarge (1x T4 16GB) lists at about $0.526 per hour and g6.xlarge (1x L4 24GB) at about $0.805 per hour, a 53 percent premium. The L4 typically delivers 2x to 3x the T4 throughput on transformer inference thanks to newer tensor cores, FP8 support, and 24GB of memory instead of 16GB. That makes the L4 roughly 25 to 50 percent cheaper per thousand inferences despite the higher hourly rate. The T4 still wins for tiny models, legacy CUDA stacks, and workloads so light that one GPU is idle most of the time, where a CPU instance or serverless may be cheaper than either.

The T4 became the default inference GPU because it was cheap, widely available, and good enough. Both parts of that reasoning are now shaky. The L4 costs about half again as much per hour and does considerably more work per hour, and the 24GB of memory it carries changes which models fit at all. For an inference fleet of any size, the comparison deserves a fresh look.

The price side

Instance	GPU	Memory	On-demand per hour	Monthly at 24x7
g4dn.xlarge (AWS)	1x T4	16GB	about $0.526	about $384
g6.xlarge (AWS)	1x L4	24GB	about $0.805	about $588
g5.xlarge (AWS)	1x A10G	24GB	about $1.006	about $734
g6e.xlarge (AWS)	1x L40S	48GB	about $1.861	about $1,359
g2-standard-4 (GCP)	1x L4	24GB	about $0.71	about $518

Monthly figures assume 730 hours of continuous running. The T4 is the cheapest GPU on the list by hourly rate, which is exactly why it keeps getting chosen without further analysis.

The throughput side

The T4 is a Turing-generation part from 2018 with roughly 65 TFLOPS of FP16 tensor throughput and 320 GB/s of memory bandwidth. The L4 is an Ada-generation part with roughly 242 TFLOPS of FP16 tensor throughput, FP8 support, and about 300 GB/s of bandwidth. On compute-bound work such as batched image classification or embedding generation, L4 throughput is often 2.5x to 3x the T4. On memory-bandwidth-bound token generation, the gap narrows to roughly 1.5x to 2x, because bandwidth is similar.

That distinction matters enormously for cost. Work out the break-even: the L4 costs 1.53x the T4, so any speedup above 1.53x makes it cheaper per request. Embedding pipelines and batch scoring, which are compute-bound and batch well, see 2.5x and land around 40 percent cheaper per unit. Single-stream chat token generation might see 1.6x and barely break even. The measurement method is the same one used in inference cost per 1,000 requests.

Worked example: 50 million embeddings per month

Say you need to embed 50 million short documents a month. On a T4 you measure 900 documents per second with a well-batched pipeline, so the job takes about 15.4 hours of GPU time per day at that volume, or about 463 GPU-hours a month. At $0.526 that is roughly $244. On an L4 you measure 2,300 documents per second, so the same volume takes about 181 GPU-hours, costing roughly $146 at $0.805. The L4 is 40 percent cheaper and finishes sooner, which also shrinks the window where you are paying for supporting infrastructure.

Now flip it. A small classifier serving 5 requests per second cannot saturate either GPU. Both instances sit mostly idle, and the bill is 730 hours times the hourly rate regardless. Here the T4 is simply cheaper by $204 a month, and a CPU instance or an autoscaled serverless endpoint may beat both, a comparison laid out in GPU versus CPU inference cost.

Memory is a hard constraint, not a preference

Model size	FP16 weights	Fits on T4 16GB?	Fits on L4 24GB?
7B parameters	about 14GB	Barely, no KV headroom	Yes, with room for KV cache
8B parameters	about 16GB	No	Yes
13B parameters	about 26GB	No	No, needs quantization
7B quantized to INT8	about 7GB	Yes	Yes, large batches

The extra 8GB on the L4 is often the deciding factor. A 7B model technically loads on a T4 but leaves almost nothing for the KV cache, which caps concurrency at one or two sequences and destroys throughput. On the L4 the same model has room for a real batch, and batch size is the single biggest lever on inference cost per request.

The fleet-level view

Individual instance rates understate the decision. A fleet of 40 inference nodes on g4dn.xlarge costs about $15,359 a month running continuously. The same 40 on g6.xlarge costs about $23,506. If the L4 delivers 2.5x the throughput, you do not need 40 of them, you need 16, which costs about $9,402 a month and saves roughly $5,957 against the T4 fleet while also cutting the number of nodes to operate, monitor, and patch. That second-order saving on operational surface area rarely appears in a spreadsheet but is worth real engineering time. Committed pricing applies to both: a one-year savings plan takes 25 to 30 percent off either fleet, so the ranking does not change, it just shifts both numbers down.

The practical rule

If your GPU is busy more than about 40 percent of the time and your workload batches, move to L4 and expect 25 to 50 percent lower cost per inference. If your GPU is mostly idle, the cheapest hourly rate wins and you should be asking whether you need a dedicated GPU at all. Either way, size the fleet from measured throughput, not from instinct, and price the change in Terraform against the resource catalog before it ships, so an instance family swap across 40 nodes does not arrive as a surprise.

FAQ

Is the L4 cheaper than the T4 for inference?

Per hour, no: about $0.805 for g6.xlarge versus about $0.526 for g4dn.xlarge on AWS, a 53 percent premium. Per inference, usually yes, because the L4 typically delivers 2x to 3x the throughput on modern models. Any speedup above 1.53x makes the L4 cheaper per request, and compute-bound work like embedding generation or batch scoring routinely exceeds that.

When should I still choose a T4?

When the GPU sits mostly idle and hourly rate dominates, when your model is small and fits comfortably in 16GB, or when a legacy CUDA or framework stack is pinned to an older architecture. At low request rates neither GPU is saturated, so the $204 per month difference at 24x7 running goes straight to the cheaper instance. Below that, consider CPU inference or a serverless endpoint instead.

How much memory difference is there between L4 and T4?

The T4 has 16GB and the L4 has 24GB. That 8GB gap decides which models are practical. A 7B parameter model in FP16 needs about 14GB of weights, which technically fits on a T4 but leaves almost no room for the KV cache, capping concurrency and throughput. On an L4 the same model has headroom for a real batch, and batch size is the biggest lever on cost per request.

What is the L4 speedup over the T4 in practice?

It depends on whether the workload is compute-bound or bandwidth-bound. Compute-bound work such as batched image classification or embedding generation commonly sees 2.5x to 3x because the L4 has roughly 242 TFLOPS of FP16 tensor throughput versus about 65 on the T4. Memory-bandwidth-bound token generation sees closer to 1.5x to 2x, since the two parts have similar bandwidth around 300 to 320 GB/s.

What does an L4 cost on GCP versus AWS?

On GCP, g2-standard-4 with one L4 lists at roughly $0.71 per hour, about $518 a month running continuously. On AWS, g6.xlarge with one L4 lists at roughly $0.805 per hour, about $588 a month. Exact rates vary by region and both clouds offer committed-use or savings-plan discounts that cut 30 to 55 percent off for steady inference fleets.

How does C3X help compare GPU instance types?

C3X prices the instance types declared in your Terraform against a live catalog, so switching a node group from g4dn to g6 shows the hourly and monthly delta in the pull request before it merges. On a fleet of 40 inference nodes that difference is thousands of dollars a month, and seeing it alongside your throughput benchmark makes the cost per inference explicit rather than inferred.

What to do next

Compare inference GPU costs before the fleet scales. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.