Designing observability retention policies: one signal at a time

Quick answer

Retention should be set per signal, not globally. Metrics are cheap and compress with downsampling, so 13 to 15 months is reasonable for capacity planning and year-over-year comparison. Traces are expensive and almost never read after two weeks, so 7 to 15 days in the hot tier. Logs sit between at 14 to 30 days hot. Audit and security records follow the compliance obligation in a cold tier. The single worst default is never-expire, which is the CloudWatch log group default and quietly accumulates $0.03 per GB-month forever.

Most organizations have one retention number. It was chosen years ago, it applies to everything, and it is simultaneously too long for traces, too short for capacity metrics, and irrelevant to the compliance requirement it was probably meant to satisfy. Retention is a per-signal decision because the value decay curve and the unit cost differ by two orders of magnitude between signals.

Value decay by signal

Signal	Peak usefulness	Long-tail use	Unit cost
Traces	Minutes to hours	Almost none	Very high
Debug-level logs	Hours	Almost none	High
Application logs	Days	Occasional investigation	High
Access logs	Days to weeks	Traffic analysis, abuse review	Medium
Metrics	Minutes to days	Capacity planning, seasonality	Low
Audit and security	Rarely read	Mandatory, audit driven	Medium

Traces and metrics sit at opposite corners. Traces are expensive per unit and useless after a fortnight. Metrics are cheap per unit and get more useful with age, because year-over-year comparison and capacity forecasting need a full seasonal cycle. Applying the same 30-day policy to both wastes money on one and destroys value on the other.

A per-signal policy that works

Signal	Hot retention	Cold retention
Traces (sampled)	7 to 15 days	90 days in Parquet if analyzed
Debug logs	3 to 7 days	None
Application logs	14 to 30 days	180 days in object storage
Access logs	30 days	13 months in object storage
Raw metrics (full resolution)	15 to 30 days	n/a
Downsampled metrics (5 min)	90 days	n/a
Downsampled metrics (1 hour)	13 to 15 months	n/a
Audit and security	90 days	Per obligation, tiered

Metric downsampling is the mechanism that makes long metric retention affordable. A series scraped every 15 seconds produces 172,800 samples per month; the same series downsampled to 1-hour resolution produces 720, a 240x reduction. Keeping 15 months of hourly data costs roughly the same as keeping 1.9 days of raw data. Prometheus-compatible backends do this with downsampling rules or built-in compaction, and it is the reason capacity planning does not need to be expensive.

The never-expire default

CloudWatch log groups are created with retention set to Never Expire unless you specify otherwise. A log group ingesting 5 GB/day accumulates 1,825 GB in a year, 5,475 GB in three years, and keeps billing $0.03 per GB-month on all of it. Across a 300-log-group estate where half were created without a retention setting, the accumulated storage routinely reaches 40 to 90 TB and $1,200 to $2,700 per month for data that nobody can find a use for.

Worse, it is invisible on a month-to-month basis because it grows a little at a time. Audit every log group for retention setting, set a sane default, and enforce it in your Terraform module so a new log group cannot be created without one. This is the highest-value one-time cleanup in the whole retention area, and it is usually a single afternoon of work.

Deriving retention from actual use

Do not guess. Every platform can report the age distribution of queried data. Pull 90 days of query history, plot the age of the oldest data touched by each query, and set hot retention at the 95th percentile. Teams consistently discover that the number is smaller than their policy: a 90-day policy where the 95th percentile query reaches back 11 days is 79 days of storage paid for out of habit.

Run the same analysis per log group rather than globally, because the distribution differs wildly. A payment service's logs may genuinely be queried at 60 days during dispute investigations while a build agent's logs are never touched after 48 hours. One policy cannot serve both without overpaying for one.

Where compliance sets the floor

Several obligations specify minimum retention, and those are inputs rather than choices. Payment card environments commonly require one year of audit history with three months immediately available. Many financial and healthcare regimes specify six or seven years for specific record types. The important design move is to separate the compliance-scoped records into their own log groups and their own storage path, so the seven-year obligation applies to the 3% of volume that is actually in scope rather than to everything.

Mixing scoped and unscoped data in one log group is a costly mistake because it forces the longest retention onto the largest volume. Splitting them is usually a matter of routing rules in the collector, and it converts a seven-year obligation on 60 GB/day into a seven-year obligation on 2 GB/day. See security log retention for the tiering detail.

Making policy enforceable

Retention decays into chaos unless it is codified. Put retention in the Terraform module that creates log groups, with a required variable and a validation block restricting it to an approved set of values. Add a policy check that fails a plan containing a log group with no retention or a retention over your maximum without an exemption tag. Review the exemptions quarterly.

Then price the result. Log groups, retention settings, metric workspaces, and archive buckets are all Terraform resources, so run the plan against the resource catalog and see what a retention change costs before it applies to three hundred log groups. Pair that with the tiering architecture so anything past hot retention lands somewhere cheap rather than being deleted outright.

FAQ

Should retention be the same for all telemetry signals?

No. Traces are expensive per unit and nearly useless after two weeks, so 7 to 15 days is right. Metrics are cheap and become more useful with age for capacity planning and seasonality, so 13 to 15 months of downsampled data is right. Logs sit between at 14 to 30 days hot. Applying one number to all of them overpays on traces and destroys value on metrics simultaneously.

How long should metrics be retained?

Thirteen to fifteen months of downsampled data, so you have a full seasonal cycle plus overlap for year-over-year comparison. Downsampling makes this cheap: a series scraped every 15 seconds produces 172,800 samples per month, while the same series at 1-hour resolution produces 720, a 240x reduction. Fifteen months of hourly data costs roughly what 1.9 days of raw data costs.

What is the most expensive retention mistake?

Leaving CloudWatch log groups at their Never Expire default. A log group ingesting 5 GB/day accumulates 1,825 GB in a year and keeps billing $0.03 per GB-month indefinitely. Across a 300-log-group estate where half were created without a retention setting, accumulated storage routinely reaches 40 to 90 TB, or $1,200 to $2,700 per month, and it grows too slowly month to month for anyone to notice.

How do I work out the right hot retention period?

Derive it from query history rather than guessing. Pull 90 days of queries, plot the age of the oldest data each one touched, and set hot retention at the 95th percentile. Do this per log group, not globally, since a payment service's logs may genuinely be queried at 60 days while build agent logs are never touched after 48 hours. Teams routinely find they are paying for 79 unused days.

How should compliance retention requirements be handled?

Separate compliance-scoped records into their own log groups and storage path, so a seven-year obligation applies to the 3% of volume actually in scope rather than to everything. Mixing scoped and unscoped data forces the longest retention onto the largest volume. Splitting is usually a routing rule in the collector, converting a seven-year obligation on 60 GB/day into one on 2 GB/day.

How does C3X help enforce retention policy?

Log groups, their retention settings, metric workspaces, and archive buckets are Terraform resources, so retention is codified rather than clicked into a console. C3X prices that estate from the plan, showing what a retention change costs before it applies to three hundred log groups, and surfacing a new log group created without an explicit retention setting while it is still a pull request.

What to do next

Codify retention and price it before it applies. C3X reads your Terraform and prices log groups, retention, and archives against a live catalog. Start with the quickstart.