Cardinality explosion: how one label multiplies your metrics bill

Quick answer

Cardinality is the number of unique label combinations on a metric, and it is multiplicative: 5 status codes times 40 endpoints times 12 regions is 2,400 time series from one metric name. Every platform bills on that count. CloudWatch charges $0.30 per custom metric per month for the first 10,000, so 2,400 series costs $720 per month; add a user ID label with 50,000 values and you get 120 million series. The rule: never put an unbounded identifier (user ID, request ID, session, email, full URL path) in a metric label. Put it in a log or a trace instead.

There is a specific line of code that turns a $400 monitoring bill into a $40,000 one, and it looks completely reasonable in review. Someone adds a label so they can break a latency metric down by customer. The metric already had labels for endpoint, method, status code, and region. Now it has one more. The diff is three characters wide and it multiplies the series count by the number of customers you have.

Cardinality is multiplication, not addition

Labels	Distinct values	Running series count
metric name only	1	1
+ status_code	5	5
+ endpoint	40	200
+ region	12	2,400
+ instance_id	250	600,000
+ customer_id	50,000	30,000,000,000

Each label multiplies rather than adds, so intuition built on additive costs fails badly here. The jump from 600,000 to 30 billion is one label. In practice you never reach the theoretical product because not every combination occurs, but you do not need to: the observed subset is still orders of magnitude larger than what you had.

What each series costs

CloudWatch bills custom metrics at $0.30 per metric per month for the first 10,000 metrics, then $0.10 for the next 240,000, $0.05 for the next 750,000, and $0.02 beyond a million. Critically, a "metric" in CloudWatch terms is a unique name plus dimension combination, so it is exactly the series count. The 2,400-series example costs 2,400 times $0.30, or $720 per month. Push to 600,000 series and you pay $3,000 for the first 10,000, $24,000 for the next 240,000, and $17,500 for the remaining 350,000, totalling $44,500 per month from a single well-intentioned dimension. See CloudWatch metrics pricing for the full tier table.

Amazon Managed Service for Prometheus prices on samples rather than series, at $0.90 per 10 million samples ingested for the first 2 billion samples per month, dropping to $0.35 per 10 million for the next 250 billion. Samples equal series times scrapes. At a 30-second scrape interval each series produces 2 samples per minute, 86,400 per month. So 600,000 series generates 51.8 billion samples per month, costing $180 for the first 2 billion plus roughly $1,743 for the remaining 49.8 billion, about $1,923 per month, plus storage at $0.03 per GB-month. Per-sample pricing is far more forgiving of cardinality than per-series pricing, which is worth knowing when you pick a backend.

Google Cloud Monitoring charges $0.2580 per MiB of ingested data for the first 100,000 MiB, then $0.1510, then $0.0610. Billing on volume rather than series count again softens the cardinality cliff, but it does not remove it, because more series means proportionally more bytes.

The self-hosted version of the bill

Running your own Prometheus does not make cardinality free, it converts the cost from an invoice line into a memory requirement. Prometheus holds roughly 1 to 3 KB of resident memory per active series once index and chunk overhead is counted. At 2 KB per series, 600,000 series needs about 1.2 GB just for the head block, and 5 million series needs 10 GB, which pushes you onto an r7g.2xlarge at roughly $0.4284 per hour, or $313 per month, before replication. Double that for a redundant pair and you are at $626 per month for one shard. The failure mode is worse than the cost: a cardinality spike OOM-kills the server during the incident you were trying to debug.

The labels that are always wrong

The unbounded set is short and memorable. User ID, customer ID, session ID, request ID, trace ID, email address, IP address, full URL path with parameters embedded, raw error message string, and timestamp. Each grows with traffic or user count rather than with your architecture, so the series count never stabilizes. If you cannot write down the maximum number of values a label can take, it does not belong on a metric.

The information is not lost, it moves. High-cardinality context belongs in logs and traces, which are priced per byte and indexed for search rather than stored as continuous time series. Wanting per-customer latency is legitimate; the right home is a trace attribute you can group by at query time, or a log field, not a metric dimension that bills continuously whether anyone looks at it or not.

Capping it before it ships

Bound what you can: bucket URL paths to route templates so /users/12345 becomes /users/:id, map status codes to classes (2xx, 4xx, 5xx) when you do not need the detail, and replace instance_id with a deployment or availability zone label when per-instance granularity is not driving any decision. In Prometheus, use metric_relabel_configs to drop labels at scrape time rather than hoping nobody adds them.

Then enforce it. Review metric definitions the way you review schema changes, because they have the same blast radius. Query your existing backend for the top series counts by metric name every month and treat anything growing superlinearly with traffic as a bug. And price the monitoring infrastructure itself from Terraform, so that a new managed Prometheus workspace or a metrics-heavy service gets costed against the resource catalog in the pull request rather than on next month's invoice.

FAQ

What is cardinality explosion in metrics?

It is the multiplicative growth in time series count when labels are added to a metric. A metric with 5 status codes, 40 endpoints, and 12 regions produces 2,400 series. Adding a 250-value instance ID label makes it 600,000, and adding a 50,000-value customer ID label makes the theoretical maximum 30 billion. Each label multiplies rather than adds, so a single innocuous-looking dimension can increase the series count by three or four orders of magnitude.

How much does high cardinality cost on CloudWatch?

CloudWatch bills each unique metric name and dimension combination as a separate custom metric at $0.30 per month for the first 10,000, $0.10 for the next 240,000, $0.05 for the next 750,000, and $0.02 beyond a million. A 2,400-series metric costs $720 per month. At 600,000 series the bill is $3,000 plus $24,000 plus $17,500, roughly $44,500 per month, all from adding one dimension to one metric.

Which labels should never go on a metric?

Any identifier whose value count grows with traffic or user count: user ID, customer ID, session ID, request ID, trace ID, email address, IP address, full URL paths with embedded parameters, raw error message strings, and timestamps. The test is whether you can state the maximum number of distinct values in advance. If you cannot, the series count will never stabilize and the metric will grow without bound.

Is per-sample pricing better than per-series pricing for cardinality?

Yes, significantly. Amazon Managed Service for Prometheus charges $0.90 per 10 million samples for the first 2 billion, then $0.35 per 10 million. At a 30-second scrape, 600,000 series produce about 51.8 billion samples per month, costing roughly $1,923 versus $44,500 for the same series count as CloudWatch custom metrics. Google Cloud Monitoring's per-MiB model is similarly forgiving. The backend's pricing unit matters enormously.

Does self-hosting Prometheus make cardinality free?

No, it converts the cost from an invoice line to a memory requirement. Prometheus uses roughly 1 to 3 KB of resident memory per active series, so 5 million series needs about 10 GB for the head block alone, pushing you to an r7g.2xlarge at roughly $313 per month, or $626 for a redundant pair. The worse failure mode is that a cardinality spike OOM-kills the server during the incident you were debugging.

How does C3X help with metrics cost?

Managed Prometheus workspaces, CloudWatch configurations, and the compute behind self-hosted monitoring are all Terraform resources. C3X prices them from the plan, so the cost of a new monitoring workspace or a larger instance class for a Prometheus server appears in the pull request. That puts a number next to the infrastructure decisions that cardinality growth eventually forces, before the growth happens.

What to do next

Keep your monitoring stack priced before it scales. C3X reads your Terraform and prices monitoring infrastructure against a live catalog. Start with the quickstart.