RAG pipeline cost explained: where the money goes in retrieval

Quick answer

A RAG pipeline spends in four places: embedding your documents and queries (per token, one-time for the corpus plus per query), a vector database to store and search embeddings (per stored vector and per query), retrieval and orchestration compute, and the LLM call that consumes the retrieved context (per input token for the augmented prompt plus output tokens). The dominant and most controllable cost is usually the LLM input tokens, because RAG stuffs retrieved context into every prompt, so retrieving less and retrieving better is the biggest lever.

Retrieval-augmented generation makes an LLM answer from your data by retrieving relevant documents and feeding them into the prompt. It is a powerful pattern with a cost that spreads across several components, and if you do not understand where the money goes, the bill grows faster than the usefulness. The good news is that a few stages dominate, so a few levers control most of the cost.

The four cost stages

Stage	Cost driver
Embedding	Per token, for the corpus and every query
Vector database	Per stored vector and per query
Retrieval compute	Orchestration and search infrastructure
LLM generation	Input tokens for context plus output tokens

Embedding turns text into vectors and is charged per token, a one-time cost for the corpus plus a small per-query cost. The vector database stores those vectors and charges for storage and search. Retrieval and orchestration run on compute. And the LLM call consumes the retrieved context, charged per input token as covered in the token cost guide, plus output tokens.

Why context size dominates

The stage that usually dominates is the LLM call, and specifically its input tokens, because RAG injects retrieved documents into the prompt on every single query. Retrieve ten large chunks and you pay for all ten tokens every time, on every request, forever. So the number and size of retrieved chunks is the biggest single cost lever in the whole pipeline, more than the database or the embeddings.

Cutting RAG cost

Retrieve less and retrieve better: return fewer, smaller, more-relevant chunks so the augmented prompt is lean, since every token in it is billed on every call. Improve retrieval quality so you do not need to stuff extra context to compensate. Cache results for repeated queries. Choose an embedding model and generation model sized to the task rather than the largest available. And right-size the vector database to your actual corpus. Each lever attacks a specific stage, and together they can cut the pipeline cost by a large factor.

Watching the whole pipeline

Because cost is spread across stages, monitor each: embedding volume, vector database size and query rate, retrieval compute, and LLM token consumption. The infrastructure pieces, the vector database and retrieval compute, are exactly what you can price before you deploy, alongside the rest of your resource catalog. Understanding the four stages keeps RAG from turning into an unbounded cost as your corpus and query volume grow.

FAQ

What are the cost components of a RAG pipeline?

Four: embedding documents and queries (charged per token, one-time for the corpus plus per query), a vector database to store and search embeddings (per stored vector and per query), retrieval and orchestration compute, and the LLM call that consumes the retrieved context (per input token for the augmented prompt plus output tokens). The LLM input tokens are usually the dominant and most controllable cost.

What is the biggest cost in a RAG pipeline?

Usually the LLM call, specifically its input tokens, because RAG injects retrieved documents into the prompt on every query. If you retrieve ten large chunks, you pay for all those tokens on every single request, so the number and size of retrieved chunks is the biggest single cost lever in the pipeline, more than the vector database or the embeddings themselves.

How do I reduce RAG cost?

Retrieve less and retrieve better: return fewer, smaller, more-relevant chunks so the augmented prompt is lean, improve retrieval quality so you need not stuff extra context, cache results for repeated queries, choose embedding and generation models sized to the task, and right-size the vector database to your actual corpus. Trimming retrieved context is the highest-impact lever since it is billed on every call.

How much does the vector database cost in RAG?

The vector database charges for storing embeddings (per stored vector, scaling with corpus size) and for querying them (per search). For large corpora it is a real cost, but it is usually smaller than the LLM input-token cost of the retrieved context. Right-sizing the database to your actual corpus and query rate, and comparing managed versus self-hosted options, keeps this stage in check.

Does retrieval quality affect cost?

Yes, indirectly and significantly. Poor retrieval tempts you to stuff more context into the prompt to compensate, which multiplies the per-call input-token cost. Better retrieval lets you send fewer, more-relevant chunks, cutting the dominant cost. So investing in retrieval quality is not just an accuracy improvement, it is one of the most effective ways to reduce the ongoing token spend of a RAG pipeline.

How does C3X help with RAG pipeline cost?

C3X prices the infrastructure stages of a RAG pipeline, the vector database, retrieval compute, and supporting services, from Terraform before you deploy, so their cost is visible in the pull request. While per-token LLM and embedding charges come from the model provider, C3X prices the cloud infrastructure that stores, retrieves, and serves your RAG workload at design time.

What to do next

Price the infrastructure behind your RAG pipeline before you scale it. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.