GPU memory and KV cache cost: why context length drives your inference bill

Quick answer

The KV cache holds attention keys and values for every token in every active sequence, and its size is 2 x layers x kv_heads x head_dim x 2 bytes per token in FP16. For a 7B model without grouped-query attention that is roughly 0.5 MB per token, so a single 8,000-token context consumes about 4 GB. On a 48 GB GPU holding 14 GB of weights, that leaves room for only about 8 concurrent long-context sequences. Since cost per request is hourly rate divided by concurrency, doubling context length roughly doubles cost per request. Grouped-query attention, cache quantization, and prefix caching are the levers that reverse it.

Most inference cost discussions stop at the model size, which is a mistake, because in production serving the KV cache frequently occupies more GPU memory than the weights do. Memory determines how many requests you can process at once, concurrency determines throughput, and throughput determines cost per request. So GPU memory accounting is inference cost accounting.

The formula

KV cache bytes per token equals 2 (for keys and values) times the number of layers times the number of key-value heads times head dimension times bytes per element. For a 7B model with 32 layers, 32 heads, and head dimension 128 in FP16, that is 2 x 32 x 32 x 128 x 2 = 524,288 bytes, about 0.5 MB per token. With grouped-query attention using 8 key-value heads instead of 32, it drops to about 0.125 MB per token, a 4x reduction.

Context length	KV per sequence, 32 KV heads	KV per sequence, 8 KV heads (GQA)
2,048 tokens	about 1.0 GB	about 0.26 GB
8,192 tokens	about 4.0 GB	about 1.0 GB
32,768 tokens	about 16.1 GB	about 4.0 GB
131,072 tokens	about 64.4 GB	about 16.1 GB

The last row is the important one. A single 128K-token conversation without grouped-query attention would need more KV cache memory than an entire 80 GB A100 has, before the weights are loaded at all.

Turning memory into cost per request

Take a 48 GB L40S at about $1.861 per hour serving a 7B model in FP16. Weights take 14 GB, and roughly 4 GB goes to activations, fragmentation, and runtime overhead, leaving about 30 GB for KV cache.

Context	KV per sequence (GQA)	Max concurrency	Relative cost per request
2,048	about 0.26 GB	about 115	1.0x
8,192	about 1.0 GB	about 30	about 3.8x
32,768	about 4.0 GB	about 7	about 16x
131,072	about 16.1 GB	about 1	about 115x

Concurrency falls roughly linearly with context length, and since the hourly rate is fixed, cost per request rises in proportion. This is why long-context features look cheap in a demo with one user and become alarming at production volume. It is the same mechanism behind the per-request arithmetic in inference cost per 1,000 requests.

The four levers

Grouped-query attention. If you control model selection, choosing an architecture with fewer key-value heads cuts KV cache by 4x or 8x with minimal quality impact. This is the single largest structural lever and it costs nothing at serving time.

KV cache quantization. Storing the cache in FP8 or int8 instead of FP16 halves or quarters it, typically with small accuracy impact for generation. That doubles or quadruples concurrency at the same memory, which directly halves or quarters cost per request.

Prefix caching. If many requests share a long system prompt or a retrieved document set, caching the computed KV for that shared prefix avoids both recomputation and duplicate storage. In a retrieval system where every request carries a 2,000-token instruction block, this can cut prefill compute by a large fraction and free significant memory, complementing the tactics in RAG infrastructure cost.

Paged attention and continuous batching. Allocating cache in small pages instead of reserving the maximum context per sequence eliminates the huge waste from sequences that finish early. Most requests use a fraction of the maximum context, so page-based allocation routinely raises effective concurrency by 2x to 4x on the same hardware.

Choose hardware on memory, then on speed

GPU	Memory	7B FP16 weights	Approx KV headroom
T4	16 GB	14 GB	almost none
L4	24 GB	14 GB	about 6 GB
L40S	48 GB	14 GB	about 30 GB
A100 80GB	80 GB	14 GB	about 60 GB

The L40S at about $1.861 per hour offers five times the KV headroom of the L4 at about $0.805, roughly 2.3 times the price. For concurrency-bound serving, that is a clear win, which is why memory capacity often beats raw compute in choosing an inference GPU.

Budget memory explicitly

Before choosing an instance type, write down weights, activation overhead, and KV cache at your target context and concurrency, and check they fit with 15 percent spare. Then price the resulting fleet in Terraform against the resource catalog. Discovering at load-test time that your context length forces a GPU three times more expensive is a far cheaper discovery than making it in production.

FAQ

How big is the KV cache?

Bytes per token equals 2 times layers times key-value heads times head dimension times bytes per element. For a 7B model with 32 layers, 32 heads, and head dimension 128 in FP16 that is about 0.5 MB per token, so an 8,192-token context uses roughly 4 GB. With grouped-query attention using 8 key-value heads it falls to about 0.125 MB per token, roughly 1 GB for the same context.

Why does context length increase inference cost?

Because KV cache grows linearly with context and competes with model weights for fixed GPU memory, which caps how many requests run concurrently. Since the hourly GPU rate is fixed, cost per request is the rate divided by concurrency. On a 48 GB GPU serving a 7B model, going from 2,048 to 32,768 tokens of context drops concurrency from about 115 to about 7, raising cost per request roughly 16 times.

How do I reduce KV cache memory?

Use a model architecture with grouped-query attention, which cuts cache size 4x to 8x with minimal quality impact. Quantize the cache to FP8 or int8 for another 2x to 4x. Cache shared prefixes so common system prompts are stored once. And use paged attention so memory is allocated in small pages rather than reserving maximum context per sequence, typically raising effective concurrency 2x to 4x.

How much GPU memory do I need for serving?

Add model weights, roughly 2 bytes per parameter in FP16, plus 3 to 5 GB of activation and runtime overhead, plus KV cache equal to per-token size times target context times target concurrency. Check the total fits with about 15 percent spare. A 7B model at 8,192 context with 30 concurrent sequences under grouped-query attention needs roughly 48 GB, which rules out 24 GB cards.

Is a bigger-memory GPU worth the higher hourly rate?

For concurrency-bound serving, usually yes. An L40S with 48 GB at about $1.861 per hour offers roughly five times the KV headroom of an L4 with 24 GB at about $0.805, for about 2.3 times the price. Since throughput scales with concurrency, the larger card often delivers lower cost per request despite the higher hourly rate.

How does C3X help with inference memory planning?

C3X prices the GPU instance types in your Terraform against a live catalog, so once you have computed a memory budget for weights plus KV cache at your target context and concurrency, the cost of the instance family that satisfies it is visible before deployment. Finding out at design time that long contexts force a more expensive GPU is far cheaper than finding out in production.

What to do next

Budget GPU memory before you budget the bill. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.