RAG infrastructure cost: the pieces behind retrieval-augmented generation

Quick answer

A retrieval-augmented generation (RAG) system's cost has four parts: the vector database storing embeddings, embedding generation (a per-token model cost to embed documents and queries), the LLM API for generation (usually the largest, driven by the context tokens you retrieve), and the retrieval/orchestration compute. The LLM and vector store usually dominate. Retrieving only the most relevant context (fewer tokens), right-sizing the vector database, and caching cut the bill.

Retrieval-augmented generation grounds an LLM's answers in your own documents by retrieving relevant context and including it in the prompt. Its cost is not one line but several, a vector database, embedding generation, the LLM API, and orchestration compute, and understanding which dominates is key to budgeting it.

The four cost components

Component	Bills for
Vector database	Storing and querying embeddings
Embedding generation	Per-token embedding of docs + queries
LLM API (generation)	Input (retrieved context) + output tokens
Retrieval/orchestration	Compute running the pipeline

You embed your documents once (and re-embed on updates) and each query, at a per-token embedding cost. The vector database stores those embeddings and serves similarity search, billed by size and query volume. The LLM API generates the answer, billed on the input tokens (the retrieved context you stuff into the prompt) plus output. Orchestration compute runs the pipeline. The LLM and vector store usually dominate.

Retrieved context drives LLM cost

A subtle but large cost driver is how much context you retrieve and include: every retrieved chunk is input tokens billed on every generation call. Retrieving ten large chunks when three would answer the question triples the input-token cost per query. So retrieving only the most relevant context, and chunking documents well, directly controls the dominant LLM cost.

Controlling RAG cost

Retrieve only the most relevant context (fewer input tokens to the LLM, the biggest lever), right-size the vector database and choose an efficient one for your scale, cache embeddings (documents do not change often, so do not re-embed unnecessarily) and cache LLM responses for repeated queries, route generation to the smallest capable model, and cap output length. Because the LLM's input tokens (retrieved context) and the vector store dominate, focus there, combining the LLM and vector database optimizations.

FAQ

What does a RAG system cost?

It has four parts: the vector database storing embeddings, embedding generation (a per-token cost to embed documents and queries), the LLM API for generation (usually the largest, driven by the retrieved context tokens), and the retrieval/orchestration compute. The LLM and vector store usually dominate, so the total depends heavily on how much context you retrieve and your query volume.

How do I reduce RAG cost?

Retrieve only the most relevant context (fewer input tokens to the LLM, the biggest lever), right-size the vector database and choose an efficient one, cache embeddings (documents do not change often) and cache LLM responses for repeated queries, route generation to the smallest capable model, and cap output length. Focus on the LLM input tokens and vector store, which dominate.

What drives LLM cost in a RAG system?

How much context you retrieve and include in the prompt. Every retrieved chunk is input tokens billed on every generation call, so retrieving ten large chunks when three would answer the question triples the input-token cost per query. Retrieving only the most relevant context, and chunking documents well, directly controls the dominant LLM cost in RAG.

Which part of RAG costs the most?

Usually the LLM API generation (driven by the retrieved-context input tokens) and the vector database, depending on scale. Embedding generation is a smaller per-token cost paid mainly when indexing documents, and orchestration compute is modest. So budget primarily for the LLM tokens and the vector store, and optimize those two first.

Should I cache in a RAG system?

Yes, in two places. Cache embeddings, since documents do not change often, so you do not re-embed unnecessarily on every index or query. And cache LLM responses for repeated or identical queries so you do not pay to regenerate the same answer. Both reduce the dominant embedding and LLM token costs for repeated content.

Does C3X estimate RAG infrastructure cost?

C3X prices the infrastructure components (the vector database, retrieval and orchestration compute) from your Terraform, so those appear before deploy. The embedding and LLM API costs are token-driven usage inputs, which you model based on document volume, query rate, and retrieved-context size to estimate the full RAG cost.

What to do next

Price your RAG infrastructure before you build it. C3X reads your Terraform and prices your resources against a live catalog. Start with the quickstart.