Compression codec cost trade offs: picking between ZSTD, Snappy, and gzip on price

Quick answer

Codec choice affects storage cost, scan cost, egress cost, and CPU cost simultaneously. Against uncompressed data, Snappy typically achieves 2.5x to 3x, ZSTD at default levels 3x to 4x, and gzip 3x to 3.5x but at far higher CPU cost. For most analytical data ZSTD is the right default: it compresses about 15 to 25 percent better than Snappy while decompressing fast enough that queries do not slow measurably. On a 200 TB lake that difference is roughly $800 a month in storage alone, plus proportional savings on every per TB scan charge and every byte of egress.

Compression is usually treated as a performance setting, chosen once by whoever set up the first pipeline and never revisited. It is actually a pricing decision that touches four separate meters: what you pay to store bytes, what you pay to scan them, what you pay to move them across a network boundary, and what you pay in compute time to compress and decompress. Optimizing any one of these in isolation gives the wrong answer.

The four meters

Meter	Typical price	Effect of better compression
Object storage	$0.023/GB-month hot	Directly proportional reduction
Per TB query scan	$5 to $6.25/TB	Directly proportional reduction
Cross region or internet egress	$0.02 to $0.09/GB	Directly proportional reduction
CPU for compress and decompress	Instance or DBU hours	Increases, sometimes sharply

The reason this gets mis optimized is that the CPU meter is the only one a developer feels directly. A slower compression step is visible in a job's runtime; a $500 a month storage difference is not visible to anyone except whoever reads the invoice. Putting both on the same page is most of the work.

Three of the four meters reward smaller output. Only CPU pushes the other way, and modern codecs have made that push much weaker than it was when gzip was the only realistic option.

Codec comparison on a 200 TB lake

Take 200 TB of uncompressed logical data written as Parquet. The table below assumes hot tier storage at $0.023 per GB per month and 400 TB of logical data scanned per month at $5 per TB.

Codec	Stored size	Storage/month	Scan bytes/month	Scan cost	Total
Uncompressed	200 TB	$4,600	400 TB	$2,000	$6,600
Snappy	~70 TB	$1,610	140 TB	$700	$2,310
ZSTD level 3	~55 TB	$1,265	110 TB	$550	$1,815
ZSTD level 9	~48 TB	$1,104	96 TB	$480	$1,584
Gzip level 6	~60 TB	$1,380	120 TB	$600	$1,980

ZSTD at its default level saves about $495 a month against Snappy, roughly $5,900 a year, on this one lake. Pushing to level 9 saves another $231 a month but raises write CPU noticeably, which matters only if you write far more than you read.

The CPU side, quantified

Decompression throughput is the number that affects query cost. Snappy decompresses fastest, ZSTD is close enough that most analytical queries show no measurable difference, and gzip is several times slower, which can genuinely extend query runtime and therefore warehouse or cluster hours. On the compression side the ordering is similar but the gaps are wider: gzip level 6 and ZSTD level 9 both cost real write time, while Snappy and ZSTD level 3 are cheap.

The practical consequence: if a job writes 2 TB a night and switching from Snappy to gzip adds 25 minutes to the run on a cluster costing $9 an hour, that is $3.75 a night, $113 a month, against $230 a month saved on storage and scanning. Marginal. Do the same comparison with ZSTD level 3, which adds perhaps 4 minutes, and the answer is obvious.

Matching codec to data temperature

Data	Recommended codec	Why
Hot, queried many times daily	ZSTD level 1 to 3	Good ratio, negligible decompression cost
Warm, queried weekly	ZSTD level 6	Storage dominates, reads are infrequent
Cold archive, rarely read	ZSTD level 12 or higher, or gzip	Only storage matters
Streaming ingestion buffers	Snappy or LZ4	Write latency dominates
Data crossing region boundaries	ZSTD, higher level	Egress at $0.02/GB rewards every byte saved

The egress row deserves emphasis. Moving 50 TB a month across regions at $0.02 per GB costs $1,000 uncompressed and about $275 with ZSTD. If you replicate data between regions, codec choice is an egress optimization as much as a storage one, and it compounds withtransfer charges between analytics services.

Two things that beat any codec

First, sorting. Compressing a column whose values are sorted or clustered produces dramatically smaller output than the same column in random order, often 2x better, because run length and dictionary encodings work properly. Sorting on your highest cardinality filter column before writing is free compression. Second, column pruning and partitioning, which reduce the bytes touched rather than the bytes stored. A codec improvement of 20 percent is worth having; a partitioning scheme that eliminates 95 percent of scans is worth an order of magnitude more. Do both, in that order of priority. C3X prices the storage, compute, and transfer resources involved from Terraform against the resource catalog.

FAQ

Which compression codec is cheapest overall?

ZSTD at default levels is the best general choice. It compresses roughly 15 to 25 percent better than Snappy while decompressing fast enough that analytical queries show no measurable slowdown. On a 200 TB lake with 400 TB scanned monthly, ZSTD level 3 costs about $1,815 a month against Snappy's $2,310 and uncompressed $6,600.

Is gzip worth using for analytics data?

Rarely. Gzip compresses slightly worse than ZSTD at default settings while decompressing several times slower, which extends query runtime and therefore warehouse or cluster hours. It remains reasonable for cold archive data that is almost never read, where only the storage meter matters and read latency is irrelevant.

Does compression reduce query cost?

Yes, directly. Per TB scan pricing charges for bytes read, and compressed data means fewer bytes. Moving 400 TB of logical monthly scanning from uncompressed to ZSTD reduces the billed scan from 400 TB to about 110 TB, cutting the charge from $2,000 to roughly $550 at $5 per TB, with no query changes at all.

How much CPU cost does better compression add?

Less than most people expect for ZSTD and a lot for gzip. If a nightly job writing 2 TB gains 25 minutes from switching to gzip on a cluster costing $9 an hour, that is about $113 a month against roughly $230 saved. ZSTD level 3 typically adds only a few minutes, making the trade clearly favourable.

Does sorting data improve compression?

Substantially, often by 2x. Sorted or clustered column values let run length and dictionary encodings work as intended, producing much smaller output than the same data in random order. Sorting on the column you filter most also narrows row group statistics, which improves pruning, so one action improves both compression and scan volume.

Should compression differ by data temperature?

Yes. Use ZSTD levels 1 to 3 for hot data queried many times a day, level 6 for warm data, and level 12 or higher for cold archive where only storage matters. Use fast codecs like Snappy or LZ4 for streaming ingestion buffers where write latency dominates, and higher levels for data crossing region boundaries where egress is charged per GB.

What to do next

Turn storage and transfer decisions into numbers. C3X prices your Terraform resources at review time. Start with the quickstart.