Tokens per dollar: benchmarking self-hosted inference economics

Quick answer

Cost per million output tokens equals the hourly instance rate divided by (tokens per second x 3,600) x 1,000,000. An L40S at about $1.861 per hour sustaining 2,500 output tokens per second under batching produces 9 million tokens an hour, about $0.207 per million. An H100 at about $12.29 sustaining 10,000 tokens per second produces 36 million an hour, about $0.341 per million. At 20 percent utilization those figures become $1.03 and $1.71. Utilization, batching, and quantization move this number more than hardware choice does.

Every inference cost argument becomes tractable once both sides are expressed in cost per million tokens. Hosted services already publish that number. Self-hosted infrastructure publishes an hourly rate instead, so you have to compute it, and the computation is where most of the interesting information lives.

The formula

Cost per million tokens equals hourly rate divided by tokens per hour, times one million. Tokens per hour is sustained tokens per second times 3,600. Sustained means under your real batch size and prompt distribution, not a single-stream microbenchmark, because the difference between those two is often a factor of ten.

Setup	Hourly	Sustained tokens/sec	$ per million output tokens
L4, 7B int8, batched	about $0.805	about 900	about $0.248
L40S, 7B FP16, batched	about $1.861	about 2,500	about $0.207
A100 80GB, 13B, batched	about $5.12	about 3,500	about $0.406
H100, 70B, batched	about $12.29	about 10,000	about $0.341
L40S, 7B, batch size 1	about $1.861	about 55	about $9.40

The last row is the point of the whole exercise. The same GPU running the same model is 45 times more expensive per token when serving one request at a time. Hardware selection moved the number by about 2x across the first four rows; batching moved it by 45x.

Utilization is the second multiplier

Sustained utilization	Effective $ per million on L40S
100 percent	about $0.207
60 percent	about $0.345
30 percent	about $0.690
10 percent	about $2.07

You pay for the hour regardless of whether tokens come out of it. A fleet sized for peak and averaging 30 percent utilization triples its real cost per token, which is usually enough to make a hosted API cheaper. That is why the honest self-hosting comparison uses effective, not peak, throughput, as discussed in GPU instance cost optimization.

Input tokens are not free either

Prefill processes the whole prompt at once and is compute-bound, so it is much faster per token than generation but not free. A 2,000-token prompt might take 60 ms of prefill on an L40S while generating 200 output tokens takes 800 ms. Prefill is therefore about 7 percent of that request's GPU time, which is roughly why hosted pricing commonly sets input tokens at a fraction of output rates. If your workload is retrieval-heavy with 8,000-token prompts and 100-token answers, prefill dominates and your cost per output token looks terrible while your cost per request is reasonable. Measure whichever unit matches your workload, and see LLM token cost explained for the pricing model behind it.

How to run the benchmark

Replay a few thousand real requests with real prompt lengths, at the concurrency you expect in production, for at least ten minutes. Record total output tokens and wall-clock time. Divide, then apply the formula. Repeat at two or three concurrency levels so you can see where throughput saturates and where latency becomes unacceptable, because the cheapest operating point is usually just below the knee of that curve.

Concurrency	Tokens/sec	p95 latency	$ per million
1	about 55	about 0.4 s	about $9.40
8	about 400	about 0.7 s	about $1.29
32	about 1,400	about 1.6 s	about $0.37
64	about 2,500	about 3.1 s	about $0.207
128	about 2,700	about 7.4 s	about $0.191

Concurrency 64 gives 93 percent of the maximum throughput at less than half the latency of 128. That is the operating point to design for, and the point at which to size the fleet.

Quantization is the cheapest throughput you can buy

Moving a 7B model from FP16 to int8 roughly halves the weight memory, from about 14 GB to about 7 GB, which frees memory for a larger batch and often raises sustained throughput by 40 to 80 percent on the same GPU. Cost per million tokens falls by the same proportion, so an L40S at about $0.207 per million can land near $0.13 with no hardware change and no additional spend. Int4 goes further at greater quality risk. Because the gain is pure margin on an instance you are already paying for, quantization is usually the first thing to test after batching and before any decision to buy different hardware.

Using the number

Once you have cost per million tokens for your setup, multiply by forecast monthly volume to get a budget, compare directly with any hosted rate to settle the build-versus-buy question, and track it over time as a health metric. A rising cost per million with flat traffic means utilization is falling or prompts are getting longer, both of which are actionable. Pair the benchmark with infrastructure priced from Terraform against the resource catalog, and you have both halves of the equation: what the fleet costs and what it produces.

FAQ

How do I calculate cost per million tokens for self-hosted inference?

Divide the hourly instance rate by sustained tokens per second times 3,600, then multiply by one million. An L40S at about $1.861 per hour sustaining 2,500 output tokens per second produces 9 million tokens an hour, about $0.207 per million. Use sustained throughput under your real batch size and prompt distribution, not a single-stream microbenchmark.

What affects tokens per dollar most?

Batching, by a wide margin. The same GPU serving one request at a time can cost 45 times more per token than the same GPU under heavy batching, roughly $9.40 per million versus $0.207 on an L40S. Utilization is the second multiplier: at 30 percent sustained utilization the effective cost triples. Hardware selection typically moves the number by only about 2x.

Do input tokens cost as much as output tokens?

No. Prefill processes the whole prompt at once and is compute-bound, so it is far faster per token than generation. A 2,000-token prompt might take 60 ms of prefill while generating 200 output tokens takes 800 ms, making prefill about 7 percent of that request's GPU time. That is why hosted pricing usually sets input rates well below output rates.

What concurrency should I run inference at?

Just below the knee of the throughput curve. In a typical L40S measurement, concurrency 64 delivers about 2,500 tokens per second at 3.1 seconds p95, while 128 delivers about 2,700 at 7.4 seconds. The higher setting buys 8 percent more throughput for more than twice the latency, so 64 is the cheaper practical operating point to size the fleet around.

How do I benchmark inference cost properly?

Replay a few thousand real requests with realistic prompt lengths at production concurrency for at least ten minutes, recording total output tokens and wall-clock time. Repeat at two or three concurrency levels to find where throughput saturates and latency degrades. That curve gives both the cheapest operating point and the throughput figure the cost formula needs.

How does C3X help with inference economics?

C3X prices the GPU instances and supporting infrastructure in your Terraform against a live catalog, giving the cost half of the tokens-per-dollar equation before anything is deployed. Combined with your measured throughput, that produces a defensible cost per million tokens and lets you compare a self-hosted fleet with any hosted rate as a straightforward arithmetic comparison.

What to do next

Turn inference budgets into arithmetic. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.