Multi-model serving cost: packing models onto fewer GPUs

Quick answer

One GPU per model is the default and usually the most expensive option. Twelve small models each on a g6.xlarge at about $0.805 per hour costs roughly $7,055 a month. Packing the same twelve onto three g6e.xlarge instances with 48GB each, at about $1.861 per hour, costs about $1,632 a month, a 77 percent reduction. The techniques are memory-based packing, GPU time-slicing, MIG partitioning on A100 and H100, and lazy loading from a shared weight cache. The constraints are total GPU memory, combined peak throughput, and the blast radius of co-tenancy.

Most organizations that serve more than a handful of models arrive at one GPU per model, because it is simple, isolated, and easy to reason about. It is also the most expensive configuration possible, since typical per-model utilization in that setup is 5 to 20 percent. The alternative is packing, and the savings are large enough that it is worth the operational complexity once you pass roughly five models.

The arithmetic of packing

Approach	Instances	Hourly	Monthly
12 models, one g6.xlarge each	12x L4 24GB	about $9.66	about $7,055
12 models on 3 g6e.xlarge	3x L40S 48GB	about $5.58	about $4,076
12 models on 2 g6e.xlarge	2x L40S 48GB	about $3.72	about $2,717
12 models on 1 g6e.12xlarge	4x L40S in one box	about $10.49	about $7,658

Note the fourth row: packing onto fewer, bigger instances is not automatically cheaper. A g6e.12xlarge bundles 4 GPUs with substantial CPU and RAM, so per-GPU it costs about $2.62 per hour against $1.861 for the single-GPU variant. Pack for GPU count, not for instance count.

Four packing techniques

Technique	Isolation	Best for	Typical density
Multi-model server process	Low	Same framework, similar models	4 to 20 per GPU
GPU time-slicing	Low	Bursty, low-QPS models	3 to 10 per GPU
MIG partitioning (A100, H100)	Hardware	Predictable per-model SLOs	Up to 7 slices
Separate GPUs, shared node	High	Large models, strict isolation	1 per GPU

Time-slicing is the easiest to adopt and gives no memory isolation, so a model that allocates too much can evict its neighbors. MIG gives hard partitions with dedicated memory and compute slices, which makes latency predictable, but partitions are fixed sizes and are only available on A100 and H100 class hardware, where the hourly rate is high enough that you need real density to benefit.

The memory budget is the binding constraint

Packing is limited by GPU memory, not by compute, in almost every real case. Each co-tenant needs weights plus activation and KV cache headroom. On a 48GB L40S, serving four 7B models quantized to int8 at about 7GB each uses 28GB of weights, leaving 20GB for caches and fragmentation, which is workable. Four 7B models in FP16 at 14GB each would need 56GB and does not fit at all. Quantization is therefore not only a latency optimization, it is what makes density possible, and density is where the money is.

Cold loading versus resident

For long-tail models that get a handful of requests an hour, keeping weights resident is wasteful even when packed. Lazy loading from a local NVMe cache, evicting least-recently-used models, lets one GPU serve dozens of models with only the hot few resident. The cost is a load penalty of 2 to 30 seconds on a cold model, which is acceptable for internal or batch-style traffic and not for interactive use. Combining a hot resident set with an LRU cold tier is how platforms serve hundreds of fine-tuned variants on a handful of GPUs, particularly when the variants are LoRA adapters of a few hundred megabytes sharing one base model.

LoRA adapters make this almost free

If your many models are fine-tuned variants of one base, serve the base once and swap adapters per request. A 7B base takes 14GB in FP16; each LoRA adapter is typically 20 to 200 MB. One 48GB GPU can hold the base plus fifty adapters with room for caches. That turns fifty dedicated endpoints, roughly $29,400 a month at g6.xlarge rates, into one instance at about $1,359 a month. The saving is over 95 percent, and it is available to anyone whose model portfolio is variants rather than genuinely distinct architectures.

What to watch

Co-tenancy means shared blast radius: one model with a memory leak or a pathological request can degrade every neighbor. Mitigate with per-model memory limits, request rate limits, and by keeping your two or three highest-value models on dedicated capacity while the long tail shares. Track per-model cost by dividing instance cost across co-tenants by their measured GPU-seconds, so chargeback stays honest, an approach consistent with showback versus chargeback. Then define the packed fleet in Terraform and price it against the resource catalog so the saving from consolidation is visible as a number before the migration is approved.

FAQ

How much does multi-model serving save?

Often 60 to 80 percent. Twelve small models each on their own g6.xlarge at about $0.805 per hour cost roughly $7,055 a month. Packed onto two or three g6e.xlarge instances with 48GB each at about $1.861 per hour, the same twelve cost about $2,717 to $4,076. The saving comes from eliminating the 5 to 20 percent per-model utilization typical of one-GPU-per-model setups.

What limits how many models fit on one GPU?

GPU memory, almost always, rather than compute. Each co-tenant needs its weights plus activation and KV cache headroom. On a 48GB L40S, four 7B models quantized to int8 at about 7GB each use 28GB and leave room for caches. The same four in FP16 at 14GB each need 56GB and do not fit, which is why quantization is what makes density possible.

What is the difference between time-slicing and MIG?

Time-slicing shares one GPU context across processes with no memory isolation, so a misbehaving model can evict its neighbors, but it is simple and works on any GPU. MIG creates hardware partitions with dedicated memory and compute on A100 and H100 class devices, giving predictable per-model latency at the cost of fixed partition sizes and limited hardware support.

How do I serve hundreds of fine-tuned model variants cheaply?

If they are LoRA adapters of a shared base model, serve the base once and swap adapters per request. A 7B base takes about 14GB in FP16 while each adapter is typically 20 to 200 MB, so a single 48GB GPU holds the base plus dozens of adapters. That replaces dozens of dedicated endpoints costing tens of thousands a month with one instance at about $1,359.

What are the risks of packing models onto shared GPUs?

Shared blast radius. One model with a memory leak or a pathological request pattern can degrade every co-tenant on the device. Mitigate with per-model memory limits and request rate limits, keep your two or three highest-value models on dedicated capacity, and let the long tail share. Also track per-model GPU-seconds so cost attribution stays accurate across tenants.

How does C3X help with multi-model serving cost?

C3X prices the instance types and counts in your Terraform against a live catalog, so consolidating twelve single-GPU nodes into three larger ones shows its monthly delta in the pull request. It also exposes the per-GPU price difference between instance shapes, which matters because packing onto fewer but larger instances is not automatically cheaper per GPU.

What to do next

See the saving from consolidation before you migrate. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.