Reducing logging and observability spend without going blind

Observability cost has an unusual property: it grows with three things at once. More traffic means more logs, more services means more metrics, and more engineers means more dashboards and more retained data. That compounding is why it quietly becomes the second largest line on many bills, and why cutting it once without changing the ingestion rules means it returns within two quarters.

Where the money actually goes

The critical asymmetry: ingestion is roughly 15 times more expensive than a month of storage on AWS. That means reducing volume matters far more than shortening retention, which is the opposite of most teams' instinct. Cutting retention from 90 days to 30 on a log group you still ingest fully saves a fraction of what dropping half the lines before ingestion saves.

Step 1: find the logs nobody reads

Query your logging platform for which log groups or streams have been searched in the last 30 days. On most estates 30 to 50 percent of ingested volume belongs to groups nobody queried once. These are usually verbose framework logs, health check endpoints, or a service that was instrumented enthusiastically and then forgotten.

For unread groups the decision is binary: either the logs matter for a future incident, in which case route them to object storage at $0.023 per GB-month instead of a queryable platform at $0.50 per GB ingested, or they do not, in which case stop emitting them. Routing to cheap storage preserves the ability to investigate while removing 95 percent of the cost.

Step 2: filter at the source, not the destination

The single most effective change is dropping lines at the collection agent before they are ingested, because ingestion is the billed event. Standard drops that rarely lose anything: DEBUG level in production, load balancer health check requests (often 20 to 40 percent of access log lines on a service with aggressive health checking), successful static asset requests, and framework startup noise.

A service logging 500 GB a month at $0.50 per GB is $250. Dropping health checks and debug lines commonly takes that to 200 GB, saving $150 a month for that one service. Multiplied across a hundred services it is the largest single lever available. The AWS specific version is in CloudWatch Logs cost optimization.

Step 3: sample success, keep every error

For high volume events where you need the shape rather than every instance, sample. Keep 100 percent of errors, warnings, and slow requests, and sample successful requests at 1 to 10 percent. Statistically you retain the ability to measure rates and latency distributions while ingesting a tenth of the bytes.

The same logic applies to tracing, where head based sampling at 1 to 5 percent is standard and tail based sampling keeps the traces that contain errors or high latency regardless of rate. The risk to name: sampling makes it impossible to find one specific customer request in the sampled out portion, so keep request IDs in logs even when the trace is dropped, and never sample audit or compliance events.

Step 4: fix metric cardinality

Custom metrics bill per unique time series, which means cost scales with the product of your label values. A metric with one label of 10 values is 10 series. Add a second label with 100 values and it is 1,000. Add a user ID or a request ID as a label and it is unbounded, which is how a single bad instrumentation line adds thousands of dollars a month.

Audit for high cardinality labels, user IDs, request IDs, full URL paths, container IDs, and replace them with bounded equivalents such as route templates and service names. Then delete metrics no dashboard or alert references, which is usually a surprising share of the total.

Step 5: tiering and commitments

Most platforms offer a cheaper tier for logs you want searchable but rarely query, and commitment tiers that discount heavily above a daily volume threshold. On Azure, moving from pay as you go Log Analytics at about $2.76 per GB to a commitment tier at 100 GB per day can cut the effective rate meaningfully, but only buy the commitment after steps 1 to 4, for exactly the same reason commitments come last everywhere else. On GCP the first 50 GiB per project per month is free before the roughly $0.50 per GB rate applies, which makes per project log routing a real lever. See GCP Cloud Logging cost.

Expected outcome and the real risk

A full pass typically cuts observability spend 40 to 70 percent. On $40,000 a month that is $16,000 to $28,000. The risk is genuine and should not be minimised: cutting the wrong signal is discovered during an incident, at the worst possible moment. Mitigate it by changing one category at a time, keeping errors and audit events at full fidelity always, and running a post change incident review that explicitly asks whether any missing data slowed the response. If it did, put that category back. The cost of one extended outage exceeds a year of observability savings.

To keep it from regrowing, make log volume a metric on the service dashboard so teams see their own ingestion, and price observability infrastructure defined in Terraform against the resource catalog before it ships.

FAQ

What to do next

Price observability infrastructure before it ships. C3X costs Terraform against a live resource catalog. Start with the quickstart.