Observability data tiering: moving telemetry to cold storage without losing it
Almost nobody queries observability data older than two weeks, yet most organizations keep months of it in the most expensive tier available. Tiering fixes that, and the price gap is roughly 100 to 1.
Quick answer
Query rates on observability data collapse with age: typically 80% of queries touch the last 24 hours, 95% the last 7 days, and under 1% anything older than 30 days. Yet storage prices differ by up to 150x across tiers, from around $0.15 per GB-month in an indexed platform to $0.023 in S3 Standard, $0.004 in Glacier Instant Retrieval, and $0.00099 in Deep Archive. Tiering 12 months of 100 GB/day from hot to a graduated cold path cuts storage from roughly $5,400 per month to under $300 per month while keeping every byte queryable.
Observability data has the steepest access decay curve of any data an engineering organization keeps. A log line is read intensely in the first hour, occasionally in the first week, and essentially never after a month, unless an auditor asks. Storage pricing, meanwhile, spans two orders of magnitude between tiers. Any time access frequency decays that fast and price varies that much, tiering is not an optimization, it is the correct architecture.
The access decay curve
| Data age | Share of queries | Latency needed |
|---|---|---|
| 0 to 24 hours | ~80% | Sub-second, live tail |
| 1 to 7 days | ~15% | Seconds |
| 7 to 30 days | ~4% | Seconds to a minute |
| 30 to 180 days | <1% | Minutes acceptable |
| Over 180 days | Rare, audit only | Hours acceptable |
If 99% of value is extracted in the first 30 days, paying premium indexed-storage prices for months 2 through 12 is paying premium for the 1%. The counter-argument is that you cannot predict which historical data an investigation will need, which is true and is why the answer is tiering rather than deletion.
The price ladder
| Tier | Price per GB-month | 36 TB (12 mo of 100 GB/day) |
|---|---|---|
| Indexed observability platform | ~$0.15 | $5,400 |
| CloudWatch Logs storage | $0.03 | $1,080 |
| S3 Standard | $0.023 | $828 |
| S3 Standard-IA | $0.0125 | $450 |
| S3 Glacier Instant Retrieval | $0.004 | $144 |
| S3 Glacier Flexible Retrieval | $0.0036 | $130 |
| S3 Glacier Deep Archive | $0.00099 | $36 |
Top to bottom is a factor of 151. Even the modest step from an indexed platform to S3 Standard is 6.5x. And the 36 TB figure assumes no compression; logs compress well, often 8:1 to 12:1, so the real archived footprint of 100 GB/day for a year is closer to 4 TB, which in Glacier Instant Retrieval costs $16 per month. The absolute numbers become almost comic once compression is applied.
A graduated policy that works
Do not move from hot to Deep Archive in one step, because you will hit an investigation that needs four-month-old data and a 12-hour restore will be unacceptable. Graduate.
| Age | Destination | Access method |
|---|---|---|
| 0 to 7 days | Indexed hot tier | Interactive query, live tail |
| 7 to 30 days | Platform standard storage | Interactive query |
| 30 to 180 days | S3 Standard-IA, Parquet | Athena, seconds to minutes |
| 180 to 395 days | Glacier Instant Retrieval | Athena with retrieval fee |
| Over 395 days | Deep Archive | Restore then query, 12 hours |
The 395-day figure is deliberate rather than 365: annual audits frequently ask for "the last full year", and having a month of margin avoids restoring from Deep Archive for a routine request. Note also that Glacier Flexible Retrieval and Deep Archive both apply a 90-day and 180-day minimum storage duration respectively, so moving data in and deleting it early incurs early-deletion charges. Only transition data you intend to keep past those minimums.
Format is half the saving
Moving raw JSON to S3 and calling it tiered leaves most of the benefit unclaimed. Converting to Parquet with partitioning by date, account, and service does three things: it compresses better (often 4x smaller than gzipped JSON), it lets Athena read only the columns a query needs, and it lets Athena skip partitions entirely.
The query economics follow. Athena charges $5.00 per TB scanned. Querying a month of raw JSON logs might scan 3 TB and cost $15. The same query over date-partitioned Parquet touching three days and four columns might scan 8 GB and cost $0.04. That 375x difference is what makes a cold tier genuinely usable rather than nominally available.
Getting the data out
The export path has its own costs. CloudWatch Logs export to S3 is free of per-GB charge for the export task itself but Kinesis Data Firehose delivery, the common streaming route, charges $0.029 per GB for the first 500 TB per month. Streaming 3,000 GB per month costs $87. Firehose format conversion to Parquet adds $0.018 per GB, another $54. Roughly $141 per month to build the archive, against $5,000+ saved.
S3 request charges matter at high object counts. PUT requests are $0.005 per 1,000, so writing 4 million small objects per month costs $20; batching into larger objects through Firehose buffering (up to 128 MB or 900 seconds) keeps object counts low and Athena performance high. Thousands of tiny files is the classic mistake that makes a data lake slow and expensive at once.
Lifecycle rules do the work
S3 lifecycle transitions are the mechanism, and they cost $0.01 per 1,000 objects transitioned to Glacier tiers. With sensibly sized objects that is negligible; with millions of tiny files it is not, a second reason to batch. Write the lifecycle policy alongside the bucket in Terraform so the tiering schedule is reviewed and version controlled rather than clicked into a console and forgotten.
A tiered archive is a handful of Terraform resources: a bucket, a lifecycle configuration, a Firehose delivery stream, a Glue catalog, and an Athena workgroup with a scan limit to prevent runaway queries. Price them from the plan against the resource catalog and compare the total against what your current retention policy is costing in the hot tier.
FAQ
How much does observability data tiering save?
Typically 95% or more on storage. Twelve months of 100 GB/day is 36 TB uncompressed, costing about $5,400 per month at an indexed platform's $0.15 per GB-month. A graduated path through S3 Standard-IA at $0.0125, Glacier Instant Retrieval at $0.004, and Deep Archive at $0.00099 brings that under $300, and compression at 8:1 to 12:1 reduces it much further still.
What tiering schedule should I use for logs?
Graduate rather than jumping straight to archive: 0 to 7 days in an indexed hot tier, 7 to 30 days in platform standard storage, 30 to 180 days in S3 Standard-IA as Parquet queried by Athena, 180 to 395 days in Glacier Instant Retrieval, and beyond 395 days in Deep Archive. Use 395 rather than 365 days so a routine annual audit request does not require a 12-hour Deep Archive restore.
Why does Parquet matter for cold observability storage?
It makes the cold tier actually usable. Parquet compresses roughly 4x better than gzipped JSON, and with partitioning by date, account, and service, Athena reads only the needed columns and partitions. A query that scans 3 TB of raw JSON at $5.00 per TB costs $15; the same query over partitioned Parquet may scan 8 GB and cost $0.04, a 375x difference.
What does it cost to build a tiered log archive?
Modest. Kinesis Data Firehose delivery charges $0.029 per GB for the first 500 TB per month, so streaming 3,000 GB costs $87, plus $0.018 per GB for Parquet format conversion, another $54. S3 PUT requests are $0.005 per 1,000 and lifecycle transitions to Glacier are $0.01 per 1,000 objects. Roughly $141 per month to build an archive that saves several thousand.
What mistakes make tiering go wrong?
Three common ones. Writing millions of tiny objects, which makes Athena slow and inflates PUT and lifecycle transition charges; batch through Firehose buffering up to 128 MB instead. Transitioning to Glacier Flexible Retrieval or Deep Archive data you will delete early, which triggers 90-day and 180-day minimum storage charges. And storing raw JSON rather than Parquet, which leaves most of the query saving unclaimed.
How does C3X help build a tiered observability archive?
The archive is Terraform: an S3 bucket, a lifecycle configuration, a Firehose delivery stream with format conversion, a Glue catalog, and an Athena workgroup with scan limits. C3X prices that estate from the plan, so you can compare the cost of building the tiered path against what your current retention settings cost in the hot tier, with real numbers on both sides before you commit.
What to do next
Price a tiered archive against your current retention bill. C3X reads your Terraform and prices buckets, streams, and lifecycle rules against a live catalog. Start with the quickstart.
Share this post
Try C3X on your own Terraform
Free and open source. No API key required. One command to install, one command to estimate.