Model checkpoint storage cost: the terabytes nobody deletes

Quick answer

A checkpoint stores weights plus optimizer states, which is roughly 12 to 16 bytes per parameter for Adam in mixed precision. A 7B model checkpoint is therefore about 84 GB, and a 70B one about 840 GB. Saving every 500 steps for a long run produces dozens of them. Fifty 7B experiments each keeping 20 checkpoints is 84 TB, about $1,932 a month on S3 Standard at $0.023 per GB-month, growing every sprint. The fix is a retention policy that keeps the best and final checkpoints only, strips optimizer states from anything archived, and lifecycles older artifacts to Glacier Instant Retrieval at $0.004 per GB-month.

Checkpoint storage is the definition of a slow leak. Each individual write feels trivial next to the GPU bill, so nobody objects, and nothing ever gets deleted because deleting a checkpoint feels risky. Two years later a team is paying four figures a month to store optimizer states from experiments whose code no longer compiles.

How big a checkpoint really is

Model	Weights FP16	Full checkpoint with Adam states	Inference-only export
1B params	about 2 GB	about 12 GB	about 2 GB
7B params	about 14 GB	about 84 GB	about 14 GB
13B params	about 26 GB	about 156 GB	about 26 GB
70B params	about 140 GB	about 840 GB	about 140 GB

The multiplier is the optimizer. Adam keeps two moment estimates per parameter plus a master copy of weights in FP32, which is why a full training checkpoint is roughly six times the inference weights. That is the single most important fact in checkpoint cost management: once a run is finished, five sixths of every retained checkpoint is dead weight.

What a research team accumulates

Scenario	Volume	S3 Standard monthly	After cleanup
One 7B run, 20 checkpoints	about 1.68 TB	about $39	about $0.64
50 experiments, 20 each	about 84 TB	about $1,932	about $32
One 70B run, 30 checkpoints	about 25 TB	about $575	about $3.22
Two years of team output	about 400 TB	about $9,200	about $200

The "after cleanup" column assumes keeping only the final inference-weights export per experiment on S3 Standard. The gap between the columns is the size of the opportunity, and it is essentially free money because nothing of value is lost.

Storage classes matter here

Class	Per GB-month	Retrieval	Use for
S3 Standard	about $0.023	free	Active run, current best
S3 Standard-IA	about $0.0125	about $0.01/GB	Recent completed runs
S3 Glacier Instant Retrieval	about $0.004	about $0.03/GB	Archive, may need quickly
S3 Glacier Deep Archive	about $0.00099	about $0.02/GB plus hours	Compliance keepsakes
EFS Standard	about $0.30	free	Never, for checkpoints

That last row is not a joke. Teams that mount a shared filesystem for convenience and write checkpoints there pay about thirteen times the S3 Standard rate, which on 84 TB is roughly $25,200 a month instead of $1,932. Shared filesystems are for active working sets, not archives. The class trade-offs generalize as in S3 storage classes compared.

A retention policy that works

During a run, keep the last three checkpoints on local NVMe (included free with p4d, p5, g6e and similar instances) for fast restart, and mirror every Nth to object storage. When a run completes, keep the best checkpoint and the final one, strip optimizer states from both to produce inference weights, and delete the rest within 7 days. Lifecycle anything older than 30 days to Glacier Instant Retrieval, and anything older than a year to Deep Archive or deletion. Tag every object with the experiment ID and owner so the policy can be automated and so the cost is attributable, consistent with tagging for cost allocation.

Do not forget request fees and versioning

Writing a sharded 840 GB checkpoint as thousands of parts generates thousands of PUT requests at about $0.005 per 1,000. That is pennies per checkpoint but real at scale, and incomplete multipart uploads left behind by failed runs are charged for storage indefinitely until a lifecycle rule aborts them. Bucket versioning is worse: with versioning on and no expiration rule, every overwritten checkpoint is retained, silently doubling or tripling storage, the trap described in S3 versioning storage cost.

Reproducibility is the real argument for keeping things

The reason teams hoard checkpoints is fear of losing the ability to reproduce a result. That fear is legitimate and the response is wrong. Reproducibility comes from the training code, the data snapshot reference, the random seed, the hyperparameters, and the final weights, all of which fit in a few hundred megabytes of metadata plus the inference export. Intermediate optimizer states from step 4,500 of a run that finished six months ago contribute nothing to reproducing it. Record the metadata rigorously in a registry and you can delete the terabytes with confidence, which converts a $9,200 monthly storage bill into a $200 one while making experiments easier to find, not harder.

Make it a default, not a cleanup project

Bake the lifecycle rules into the bucket definition in Terraform so every new experiment bucket inherits them, and price the storage layer against the resource catalog before it is created. A retention policy written into infrastructure code applies automatically and forever; a cleanup project happens once and then the terabytes come back.

FAQ

How large is a model checkpoint?

Roughly 12 to 16 bytes per parameter for a full training checkpoint using Adam in mixed precision, because the optimizer keeps two moment estimates per parameter plus an FP32 master copy of weights. A 7B model checkpoint is about 84 GB, a 13B is about 156 GB, and a 70B is about 840 GB. Inference-only exports are about six times smaller, around 14 GB for a 7B model.

How much does checkpoint storage cost?

On S3 Standard at about $0.023 per GB-month, one 7B run keeping 20 checkpoints is about 1.68 TB, roughly $39 a month. Fifty such experiments reach about 84 TB and roughly $1,932 a month. Two years of accumulated team output commonly passes 400 TB, around $9,200 a month, almost all of which is optimizer states nobody will ever load again.

What checkpoint retention policy should I use?

During a run keep the last three checkpoints on included local NVMe for fast restart and mirror every Nth to object storage. On completion, keep the best and final checkpoints, strip optimizer states to produce inference weights, and delete the rest within seven days. Lifecycle anything over 30 days to Glacier Instant Retrieval at about $0.004 per GB-month and expire or deep-archive after a year.

Should I store checkpoints on a shared filesystem?

Not for archives. EFS Standard costs about $0.30 per GB-month against $0.023 for S3 Standard, roughly thirteen times more, so 84 TB of checkpoints would cost about $25,200 a month instead of $1,932. Shared filesystems are appropriate for active working sets during a run, but completed checkpoints belong in object storage with lifecycle rules.

What hidden checkpoint storage costs exist?

Incomplete multipart uploads from failed runs are stored and billed indefinitely unless a lifecycle rule aborts them. Bucket versioning with no expiration rule retains every overwritten checkpoint, silently doubling or tripling storage. And writing a sharded 840 GB checkpoint as thousands of parts generates thousands of PUT requests at about $0.005 per 1,000, which is small per run but real across a campaign.

How does C3X help with checkpoint storage cost?

C3X prices buckets, storage classes, and volumes declared in Terraform against a live catalog, so the cost of a new experiment storage layer is visible before it is created. Because lifecycle rules are also Terraform, baking retention into the module means every future experiment bucket inherits it automatically rather than relying on a periodic cleanup project.

What to do next

Set checkpoint retention in code, not in a cleanup sprint. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.