Security log retention cost: paying for a year of data nobody queries

Security telemetry has an unusual shape. The retention requirement is long, often 12 months and sometimes seven years. The query rate is extremely low: outside of an active investigation or an audit, almost nobody touches data older than a few weeks. And the volume is high, because you are keeping authentication events, network flow records, API calls, DNS queries, and endpoint telemetry from every system you own. Long retention plus low query rate plus high volume is precisely the profile that tiering was invented for.

What security telemetry actually costs in a hot tier

That is ingestion only. Add retention: 12 months of 60 GB/day accumulates 21,900 GB. At a SIEM retention price of $0.10 to $0.20 per GB-month, the stored volume alone costs $2,190 to $4,380 per month by the end of year one, and the ingestion charge repeats every month. The all-in figure for a fully hot 12-month security data estate routinely exceeds $8,000 to $12,000 per month for a mid-sized organization, and considerably more once flow log volume grows with the network.

The tiering model

The arithmetic is not subtle. Holding the 180-to-365-day slice in Glacier Instant Retrieval costs $44 per month; holding the same slice in a hot SIEM tier costs roughly $1,665. The 38x difference is why every mature security data architecture has a cold tier, and why the ones that do not are the ones with six-figure annual line items nobody can justify.

Query economics in the cold tier

The objection is always "but we need to search it during an investigation". You do, occasionally, and it is affordable. Athena charges $5.00 per TB scanned. Querying the full 11.1 TB cold slice costs $55.50, and with Parquet formatting plus date and account partitioning a realistic investigative query scans perhaps 200 GB and costs $1.00. Even ten investigations a month touching a terabyte each is $50.

Glacier Instant Retrieval adds a retrieval charge of $0.03 per GB, so scanning 200 GB adds $6. Glacier Deep Archive requires a restore taking 12 hours at $0.02 per GB for bulk, which is fine for a compliance audit and wrong for an active incident. That is the whole reason the 180-to-365-day slice sits in Instant Retrieval rather than Deep Archive: you are buying the option to investigate at normal speed.

Storage format matters more than most teams expect. Converting JSON security logs to Parquet with appropriate partitioning typically reduces scanned bytes by 80% to 95% for selective queries, because Athena reads only the needed columns and partitions. That turns $55 full-table scans into $2 ones, which changes the cold tier from theoretically queryable to routinely queryable.

What must stay hot

Not everything can be tiered. Real-time detection rules need to evaluate events as they arrive, so the detection pipeline consumes the live stream regardless of where the data lands afterwards. Correlation rules with a 30-day lookback need 30 days accessible at low latency. Threat hunting workflows generally work over 90 days. Beyond that, the access pattern is audit and investigation, which tolerates seconds instead of milliseconds.

So the design is: stream everything through the detection engine, index 30 to 90 days in the hot tier based on your correlation window, and land the full stream in object storage in Parquet from day one so the archive is complete and queryable without a re-ingestion step. Costs for SIEM platforms vary widely, but this architecture is platform-independent.

Reducing the volume itself

Before tiering, trim. VPC Flow Logs in the default format include fields most detections never use, and switching to a custom format with only the needed fields cuts volume 30% to 40%. Aggregating flow logs at a 10-minute interval instead of 1 minute cuts records by up to 90% for steady connections, at the cost of timing granularity. CloudTrail data events for high-traffic S3 buckets can be scoped to sensitive prefixes rather than every object. Each of these removes cost at the most expensive meter.

Flow log configurations, CloudTrail trails, S3 lifecycle rules, and Glue catalogs are all Terraform, so the tiering architecture is code you can price. Run the plan against the resource catalog and the difference between a fully hot estate and a tiered one shows up as a number before you commit to twelve months of it.

FAQ

What to do next

Design security log tiering with real prices. C3X reads your Terraform and prices buckets, trails, and lifecycle rules against a live catalog. Start with the quickstart.