Dashboards and queries: the observability cost that runs while nobody watches

Everyone understands that ingesting data costs money. Far fewer teams track what it costs to look at the data, and on log platforms that second meter can exceed the first. A dashboard is not a passive view. It is a scheduled query that executes on a timer, in the background, in every open browser tab, whether or not a human is reading it.

The query meters

The GetMetricData row deserves a moment. At $0.01 per 1,000 metrics requested, a dashboard with 40 widgets each pulling 25 metrics requests 1,000 metrics per refresh, costing $0.01 each time. On a 10-second refresh that is 259,200 refreshes per month and $2,592 for a single dashboard. This is a real and frequently encountered bill, and it comes entirely from a refresh-interval dropdown.

Log scan arithmetic

Logs Insights bills on bytes scanned, and the scan is determined by the time range and the log groups selected, not by how selective your filter is. A filter that matches three lines still scans every byte in the window.

Now put that on a refresh timer. A dashboard widget running a 7-day Logs Insights query on a 5-minute refresh executes 8,640 times per month at $3.50, which is $30,240 per month for one widget. Teams do discover this, usually about six weeks in, when the CloudWatch line on the bill overtakes the compute line. It is the single most common source of a surprise monitoring bill.

The structural fix

Dashboards should never scan raw logs. If a number belongs on a dashboard, it belongs in a metric. CloudWatch metric filters extract values from logs at ingestion time and publish them as metrics at $0.30 per metric-month, after which the dashboard reads the metric instead of rescanning the logs. A widget that cost $30,240 per month as a log query costs $0.30 per month as a metric plus a fraction of a cent in GetMetricData calls.

The same logic applies with Prometheus: use recording rules to precompute expensive aggregations rather than running the raw query on every dashboard refresh. A recording rule evaluates once per interval and stores a small series; the dashboard then reads a cheap series instead of processing billions of samples on each load.

The unwatched dashboard problem

Dashboards accumulate. A team creates one per incident, per service, per experiment, and rarely deletes any. Typical audits find that 40% to 60% of dashboards have not been opened in 90 days. At $3.00 per CloudWatch dashboard-month, 120 dashboards cost $351 per month with perhaps 60 of them dead, which is small. The problem is that dead dashboards with log-scan widgets still cost query charges if anything loads them, including embedded views, TV wallboards left on in an empty office, and browser tabs pinned since March.

The wallboard case is worth naming specifically. A screen in a corridor running a dashboard on a 10-second refresh, 24 hours a day, executes 259,200 queries a month. If any of those widgets scan logs, the corridor screen is the most expensive display in the building. Set wallboards to a 1 or 5 minute refresh and make sure their widgets read metrics.

Controls that actually work

The Athena workgroup limit is underused and excellent. Setting a per-query data scan limit of, say, 100 GB means a badly written unpartitioned query fails at $0.50 instead of succeeding at $500. It converts a cost incident into an error message, which is the right trade.

Dashboards, metric filters, Athena workgroups with their scan limits, and Grafana workspaces are all Terraform resources. Price them from the plan against the resource catalog, and pair that with a review habit that asks of every new widget whether it reads a metric or scans a log. That single question, asked in code review, eliminates almost all query-cost incidents before they happen.

FAQ

What to do next

Keep dashboards from quietly billing. C3X reads your Terraform and prices dashboards, filters, and workgroups against a live catalog. Start with the quickstart.