Building a cost data pipeline on the AWS CUR

Quick answer

The AWS Cost and Usage Report (CUR) is line-item billing data delivered to S3, granular down to the hour and resource. Build a pipeline that lands the CUR in S3, catalogs it (AWS Glue), and queries it with Athena or loads it into a warehouse, then layers allocation logic (tag-based attribution, shared-cost splits, amortized commitments) on top. This gives you cost analysis far beyond Cost Explorer: custom allocation, unit-economic joins to business data, and anomaly queries. The reason to build it is control: the CUR is the ground truth, and a pipeline turns it into the allocated, business-joined view your own FinOps questions need, which no packaged dashboard fully answers.

AWS Cost Explorer is convenient but limited: preset dimensions, no joins to your business data, and allocation logic you cannot customize. When your FinOps questions outgrow it, cost per tenant joined to your customer table, a shared-cost split with your own rule, amortized commitments allocated your way, you need the raw data. That raw data is the Cost and Usage Report, the most detailed billing information AWS produces, and building a pipeline on it is how mature FinOps teams get the exact view they need.

What the CUR is

The CUR is a set of line-item files delivered to an S3 bucket on a schedule, with one row per resource per usage type per hour (or per day). Each row carries the account, service, usage type, cost, pricing, and your resource tags as columns. It is comprehensive and correspondingly large, millions of rows a day for a big account, which is exactly why you do not read it by hand. You process it. It is the same ground-truth billing export that finding untagged resources and budget tracking both draw on.

The pipeline architecture

Stage	Component
Land	CUR delivered to an S3 bucket
Catalog	AWS Glue crawler builds the table schema
Query	Athena over S3, or load into a warehouse
Transform	Allocation, shared-cost splits, amortization
Serve	Dashboards, reports, alerts, business joins

The standard pattern lands the CUR in S3, uses a Glue crawler to catalog its schema, and queries it with Athena, which reads the S3 files directly with SQL and no server to run. For heavier or repeated analysis, teams load the CUR into a warehouse like Redshift or Snowflake. Athena is the low-effort start: point it at the CUR, write SQL, pay per query scanned. Partition the data by date and use a columnar format (Parquet) so queries scan less and cost less, the same efficiency that governs any data pipeline.

The allocation layer is the point

Raw CUR rows are not yet allocated cost, and this transform layer is where the pipeline earns its keep. Here you apply your own logic: attribute each row to a team by its tags, split shared costs by your chosen method, amortize reserved-instance and savings-plan charges across the usage they cover so a big upfront payment does not distort a single month, and blend commitment discounts. None of this is available your way in a packaged tool. The output is allocated cost that matches how your organization actually reports.

Joining to business data

The highest-value queries join cost to your own metrics. With the CUR in a warehouse next to your customer, request, or tenant tables, you can compute cost per tenant, cost per customer, or cost per request directly, the unit economics that no cost tool can produce because it does not have your business data. This is the real reason to build the pipeline: it is the only place your cloud cost and your business metrics live together, and that join is where FinOps insight comes from.

Keeping the pipeline sane

A CUR pipeline is itself infrastructure with a cost, S3 storage, Athena scan charges, warehouse compute, so keep it efficient: partition and compress the data, expire old CUR versions with a lifecycle policy, and avoid full-table scans. Watch that your cost-analysis pipeline does not become a noticeable line on the bill it is analyzing. And price the pipeline's own infrastructure against the resource catalog before you build it, so the tool you use to understand cost is itself understood, closing the loop on your own data platform.

FAQ

What is the AWS Cost and Usage Report?

The CUR is the most detailed billing data AWS produces: line-item files delivered to an S3 bucket on a schedule, with one row per resource per usage type per hour or day. Each row carries the account, service, usage type, cost, pricing, and your resource tags. It is comprehensive and very large, millions of rows a day for a big account, so it is meant to be processed by a pipeline rather than read by hand.

How do I build a pipeline on the CUR?

Land the CUR in S3, use an AWS Glue crawler to catalog its schema, and query it with Athena (SQL directly over S3, no server) or load it into a warehouse like Redshift or Snowflake for heavier analysis. Then add a transform layer for allocation, shared-cost splits, and commitment amortization, and serve it to dashboards and business joins. Athena is the low-effort start; a warehouse suits repeated, heavy analysis.

Why build a CUR pipeline instead of using Cost Explorer?

Because Cost Explorer has preset dimensions, no joins to your business data, and allocation logic you cannot customize. A CUR pipeline gives you the raw ground-truth data so you can apply your own allocation rules, amortize commitments your way, and, most importantly, join cost to your customer, request, or tenant tables to compute unit economics. Those custom, business-joined views are exactly what packaged dashboards cannot produce.

What is the allocation layer in a CUR pipeline?

It is the transform stage where raw CUR rows become allocated cost: attributing each row to a team by its tags, splitting shared costs by your chosen method, amortizing reserved-instance and savings-plan charges across the usage they cover so a big upfront payment does not distort one month, and blending commitment discounts. This logic is applied your way, matching how your organization reports, which no packaged tool offers out of the box.

How do I keep a CUR pipeline from getting expensive?

Partition the data by date and store it in a columnar compressed format like Parquet so Athena scans less per query, expire old CUR versions with an S3 lifecycle policy, and avoid full-table scans. A cost-analysis pipeline is itself infrastructure with S3 storage, Athena scan charges, and warehouse compute, so watch that the tool analyzing your bill does not become a noticeable line on it. Price its infrastructure before building.

How does C3X complement a CUR pipeline?

A CUR pipeline analyzes cost after it happens; C3X prices infrastructure from Terraform before it deploys. Together they cover both directions: the pipeline gives you allocated historical cost joined to business metrics, and C3X gives you the forward-looking cost of a change in the pull request. You can also price the pipeline's own infrastructure with C3X, so the platform you use to understand cost is itself understood before you build it.

What to do next

Understand cost before it happens, not just after. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.