Distributed tracing at scale: what span volume does to your bill

Quick answer

Span count, not request count, drives tracing cost, and spans grow with architectural depth. A request crossing 4 services with database and cache instrumentation produces 18 to 35 spans; adding a service mesh adds 2 spans per hop. At 500 million requests per month and 25 spans each, that is 12.5 billion spans. At a typical $2.00 per million spans, the bill is $25,000 per month. Splitting a service into three microservices can raise tracing cost 40% with no traffic change. Control it by sampling hard, limiting span attributes, and not instrumenting every internal function call.

Tracing is the observability signal whose cost is least correlated with user-visible scale. Doubling traffic doubles your trace bill, which is expected. Splitting a monolith into six services also roughly doubles it, with identical traffic, and nobody put that on the migration plan. Understanding the span multiplier is the whole game.

Where spans come from

Source	Spans per request
Ingress / API gateway	1 to 2
Each service hop (server + client)	2 per hop
Service mesh sidecars	2 per hop
Database calls	1 per query, often 3 to 10
Cache operations	1 per operation, often 2 to 8
Outbound HTTP calls	1 each
Message queue publish/consume	2 per message
Auto-instrumented internal functions	0 to hundreds

The last row is where things go wrong. Some auto-instrumentation libraries, especially in verbose configurations, create a span for every ORM operation, every serialization step, and every middleware layer. A single request can emit 200 spans, 95% of which nobody has ever looked at. Audit the span names appearing in your backend and count how many you have ever filtered on.

The architectural multiplier, priced

Take 500 million requests per month. Under a monolith with database instrumentation, roughly 9 spans per request gives 4.5 billion spans. Under six microservices with a mesh and the same database work, roughly 25 spans per request gives 12.5 billion.

Architecture	Spans/month	At $2.00/M spans	At $0.20/M (Cloud Trace)
Monolith, 9 spans/req	4.5B	$9,000	$900
4 services, 18 spans/req	9.0B	$18,000	$1,800
6 services + mesh, 25 spans/req	12.5B	$25,000	$2,500
Verbose auto-instrumentation, 90 spans/req	45B	$90,000	$9,000

Those numbers assume 100% sampling, which nobody at this scale should run. At 5% head sampling the top row becomes $450 and the bottom becomes $4,500, which is why sampling economics is the first lever. But note that sampling is a multiplier applied after the span multiplier, so an architecture emitting 90 spans per request is still 10x more expensive than one emitting 9, at any sampling rate.

Attributes: the second dimension

Backends that bill on ingested bytes rather than span count care about attribute payload. A span with 8 attributes might be 400 bytes; the same span with full HTTP headers, SQL statements, request bodies, and a 30-field resource block can reach 4 KB. That tenfold difference applies to every span you keep.

Typical offenders are the complete SQL statement including parameters (which is also a data-exposure problem), full request and response bodies, environment variable dumps in resource attributes, and redundant resource attributes repeated on every span when they belong once on the resource. Strip these in the collector with a transform processor rather than asking forty teams to change instrumentation.

Storage and retention at span scale

Twelve and a half billion spans at 600 bytes each is 7.5 TB per month raw, perhaps 1.5 TB compressed. In a managed backend you pay whatever retention they charge; self-managed in S3 Standard at $0.023 per GB-month that is $35 per month per month retained, which is trivial. The expensive part of tracing has never been storage, it is ingestion and indexing.

This asymmetry argues for the same tiering pattern used elsewhere: keep 7 to 15 days in the indexed backend where trace-by-ID lookup and service maps work, and write the full sampled stream to object storage in Parquet for historical analysis at $5.00 per TB scanned with Athena. Aggregate analysis over six months of traces then costs a few dollars per query instead of requiring six months of hot retention.

Practical reductions, ranked

Action	Typical span reduction
Tail sampling, keep errors and slow traces	95%+
Disable verbose auto-instrumentation	60% to 90%
Drop mesh sidecar spans where app spans exist	~40%
Batch ORM spans into one database span	20% to 50%
Strip large attributes in the collector	50% to 80% of bytes
Skip tracing for internal health and admin paths	5% to 20%

The mesh row deserves a note. Running both application-level and sidecar-level tracing produces two spans describing the same hop from slightly different vantage points. Both are occasionally useful; neither is usually worth 40% of your span budget. Pick one as the default and enable the other selectively.

Budgeting tracing before an architecture change

The actionable insight is that tracing cost should be a line item in any service-decomposition proposal. Splitting one service into three adds roughly 4 spans per request through that path, plus mesh spans if you run one. At 500 million requests per month and $2.00 per million spans, 4 extra spans is 2 billion spans and $4,000 per month at full sampling, or $200 at 5%. Not huge, but it should be on the page next to the compute estimate.

The services, mesh, collector tier, and trace storage are all Terraform, so price the architecture from the plan against the resource catalog and let the decomposition proposal carry a real observability number rather than an assumption that tracing is free.

FAQ

What drives distributed tracing cost at scale?

Span count, which is traffic times architectural depth times instrumentation verbosity. A request crossing four services with database and cache instrumentation produces 18 to 35 spans, and a service mesh adds 2 per hop. At 500 million requests per month and 25 spans each, that is 12.5 billion spans, costing roughly $25,000 per month at $2.00 per million spans before any sampling is applied.

Does splitting a monolith increase tracing cost?

Yes, substantially, with no traffic change. A monolith with database instrumentation might emit 9 spans per request; six microservices behind a mesh doing the same work emit around 25. At 500 million requests per month that is 4.5 billion spans versus 12.5 billion, or $9,000 versus $25,000 per month at $2.00 per million. Tracing cost belongs as a line item in any decomposition proposal.

What is the biggest source of wasted spans?

Verbose auto-instrumentation. Some libraries create a span for every ORM operation, serialization step, and middleware layer, pushing a single request to 200 spans where 95% are never inspected. Audit the span names appearing in your backend and count how many you have ever filtered on. Disabling verbose auto-instrumentation typically cuts span volume 60% to 90% with no loss of debugging capability.

How do span attributes affect tracing cost?

On byte-billed backends, enormously. A span with 8 attributes might be 400 bytes; the same span carrying full HTTP headers, SQL statements with parameters, request bodies, and a 30-field resource block can reach 4 KB, a tenfold difference applied to every span kept. Strip these with a collector transform processor rather than coordinating instrumentation changes across every team.

Should I keep traces in the hot backend for months?

No. Ingestion and indexing dominate tracing cost, while storage is cheap: 12.5 billion spans at 600 bytes is about 1.5 TB compressed, roughly $35 per month in S3 Standard. Keep 7 to 15 days in the indexed backend where trace lookup and service maps work, and write the sampled stream to Parquet in object storage for historical analysis at $5.00 per TB scanned with Athena.

How does C3X help with tracing cost?

The services, service mesh, collector tier, and trace storage behind a tracing architecture are all Terraform resources. C3X prices them from the plan, so a service-decomposition proposal carries a concrete infrastructure number alongside the span-volume implications, rather than treating tracing as a free side effect of splitting a service into three.

What to do next

Put a number on the tracing implications of an architecture change. C3X reads your Terraform and prices services, meshes, and storage against a live catalog. Start with the quickstart.