Telemetry egress cost: what shipping logs and metrics out costs

Observability pipelines are one of the few workloads that move data continuously, in one direction, out of your cloud, forever. Every log line, every metric data point, every span. The vendor bill for ingesting it is usually the number that gets attention, but the transfer to get it there is a separate charge on your cloud bill, and it is charged at the most expensive rate the cloud has.

The volumes involved

Traces dominate at full sampling because each span carries substantial metadata, and a single request through a microservice architecture can produce 20 to 50 spans. This is why nobody samples traces at 100 percent in production at scale.

Compression is the first and biggest win

Structured logs are extremely repetitive: the same field names, the same service identifiers, the same message templates over and over. Gzip on a batch of JSON log lines typically achieves 8:1, and zstd at a moderate level often reaches 10:1 to 12:1 with lower CPU cost. A 10 TB monthly log volume compressed 10:1 ships 1 TB, taking egress from about 920 dollars to about 92 dollars.

Most modern collectors compress by default, but not all do, and some disable it when using certain protocols. It is worth verifying rather than assuming. The general principle is covered in compression to reduce cloud cost.

Sampling and filtering at the source

The second lever is not sending data at all. Tail-based sampling keeps traces that are slow or erroneous and drops the boring majority, commonly retaining 1 to 5 percent of spans while preserving nearly all diagnostic value. On the 25 TB trace example, 2 percent sampling takes egress from about 2,300 dollars to about 46 dollars.

For logs, dropping debug and trace levels in production is usually safe and often removes 60 to 80 percent of volume. Health check access logs, which are high-volume and near-zero-information, are another easy removal: a service receiving a health probe every 5 seconds from 3 load balancer nodes generates 1.5 million log lines per month per instance for no reason. Filtering at the collector rather than at the vendor means you avoid both the egress and the ingest charge.

The vendor ingest charge dwarfs the egress

It is worth keeping proportion. Log ingest at typical vendor list pricing runs from roughly 0.25 to 2.50 dollars per GB depending on retention and indexing, so the 10 TB example might cost 2,500 to 25,000 dollars per month at the vendor against 920 dollars in egress. The good news is that every lever that reduces egress reduces ingest by the same proportion, usually by more, since vendors bill on uncompressed volume. Cutting 60 percent of log volume saves 60 percent of both numbers.

Keeping telemetry in-region

A third approach is to not export at all. Writing logs to CloudWatch Logs costs 0.50 dollars per GB ingested and 0.03 dollars per GB-month stored, with no egress charge since it stays in-region. For 10 TB that is 5,000 dollars, which is worse than many vendor combinations, so this is not automatically cheaper. But routing high-volume, low-value telemetry to object storage in-region at 0.023 dollars per GB-month, queryable on demand, and sending only curated signals to the vendor is a pattern that works well. A 10 TB month costs 230 dollars in S3 storage and nothing in transfer.

The NAT gateway trap

If collectors run in private subnets and reach the vendor through a NAT gateway, every telemetry byte pays 0.045 dollars per GB in data processing on top of the 0.09 dollars per GB in egress, making the effective rate 0.135 dollars per GB. On 40 TB per month that is an extra 1,800 dollars for a configuration detail. Routing collectors through a subnet with a direct internet gateway path, or aggregating through a small number of egress nodes, avoids most of it.

A sensible pipeline

Compress everything at the agent with zstd. Filter debug logs and health checks at the collector. Sample traces tail-based at 1 to 5 percent, keeping all errors. Aggregate metrics at the edge so you ship rollups rather than raw series where cardinality is high. Archive raw data to in-region object storage for the rare deep investigation. Send only what you actually query to the vendor. Done properly this cuts both the egress and the ingest bill by 80 percent or more. Price your observability infrastructure against the resource catalog.

FAQ

What to do next

Price your observability pipeline before it ships terabytes. C3X costs Terraform against a live resource catalog. Start with the quickstart.