RAG in Production: What the Tutorials Don't Cover
Most RAG tutorials get you to a working demo in an afternoon. The jump from that demo to a system that reliably serves enterprise users at scale is where most implementations quietly fail. This is what that jump actually involves.
A production RAG system rarely looks like the first version anyone built. The first architecture usually wasn't foolish. The problems that matter just don't show up until you have real users asking real questions of messy documents.
What follows is the gap between the tutorial and what we actually run. If you are evaluating a RAG implementation for enterprise use, whether you build it or buy it, these are the things to ask about.
Your embedding model is the last thing that matters
Every conversation about RAG eventually becomes a conversation about which embedding model to use. OpenAI's ada-002 vs text-embedding-3-large vs a self-hosted Cohere model. It is the wrong conversation to be having first.
Embedding model quality matters, but less than people think. Most of your retrieval quality comes from chunking strategy, metadata design, query preprocessing, and whether you run hybrid search or pure semantic. A mediocre embedding model on well-chunked documents will usually beat the best model available on badly chunked ones.
Pick a reasonable embedding model early (text-embedding-3-small works fine for most English enterprise content, and Cohere's multilingual-v3 if you need Arabic) and spend your time on the parts of the pipeline that actually determine whether users get good answers.
Chunking strategy is where you win or lose
The default chunking approach in every tutorial is fixed-size chunks with some overlap. Split at 512 tokens, 50-token overlap, done. This works well enough on the tutorial dataset and falls apart on real enterprise documents for a simple reason: enterprise documents are not uniformly structured.
A legal contract has dense definitional clauses followed by pages of boilerplate followed by schedules that reference the clauses. A technical manual has an overview section, detailed procedures, a troubleshooting section, and appendices. Fixed-size chunking cuts across these structures arbitrarily, splitting a clause mid-sentence or grouping a troubleshooting step with unrelated content from the next section.
What we use instead: structural chunking that respects the document's own hierarchy. For documents with clear heading structure, chunk at the section level, not the token level. For long sections, use semantic chunking that breaks on topic shifts rather than token counts. Extract tables and structured data separately. A table embedded in a 512-token chunk loses most of its information when flattened into text.
Our take
The chunking approach needs to be designed per document type, not once globally. A policy document and an invoice are different objects and need different treatment.
Pure semantic search is not enough. Hybrid search almost always is.
Semantic search finds documents with similar meaning. Keyword search finds documents with specific terms. Enterprise queries need both.
A user searching for "section 4.2 of the procurement policy" needs keyword matching. Semantic search will find documents about procurement policies in general and miss that specific section. A user searching for "what happens if a supplier delivers late" needs semantic search, because the document probably uses words like "delivery delay" or "SLA breach" rather than the exact query terms.
We run hybrid search on almost everything now: BM25 for keyword matching, vector search for cosine similarity, and a reciprocal rank fusion step to combine the results. Measure it against a human-rated eval set and hybrid usually wins over pure semantic search, because real enterprise queries mix exact terms with plain-language questions.
If you are evaluating a RAG vendor and they are running pure vector search, ask why. The answer might be legitimate — there are cases where document structure makes hybrid search less valuable. But "we haven't implemented it yet" is a common reason.
Metadata belongs in the design from day one
Every chunk in a production RAG system should carry structured metadata: document type, source system, date, author, department, access permissions, version, and whatever domain-specific fields are relevant. This metadata does two things that retrieval alone cannot.
First, it enables filtered retrieval. A user in the legal team asking a question should get answers from legal documents, not from every department's documentation. "What is our policy on contractor payments" should retrieve from the legal and finance document libraries, filtered by access permissions, not from across the entire corpus including IT policies and HR handbooks.
Second, it enables cited answers. A user asking about a policy wants to know which version of the policy they are reading and when it was last updated. Without document-level metadata attached to every chunk, the answer "according to the procurement policy" tells the user nothing they can act on. With metadata, the system can say "Section 4, Procurement Policy v2.3, updated March 2026," and the user can act on that.
The freshness problem will surprise you at the worst time
RAG systems have a freshness problem that nobody designs for until it causes a real incident. A document gets updated in the source system. The RAG index still contains the old version. Users start getting answers based on superseded information, and there is no warning that the retrieval results are stale.
Picture a compliance policy update that takes a week to reach the RAG index because re-indexing runs on a weekly schedule. For that week, users get answers based on the old policy, and nothing tells them so. The first person to notice may be an auditor.
The fix is not complicated: implement a document change listener that triggers re-indexing within hours of a document update, not on a weekly schedule. Attach an "indexed at" timestamp to every chunk. Surface that timestamp in the UI so users can see how recent their answer is. For high-stakes content such as policies and contracts, consider treating any answer based on a chunk older than 30 days as requiring a freshness warning.
Nobody wants to build the freshness pipeline. It is unglamorous infrastructure. Build it anyway.
The evaluation set you build in week one will save you in month six
Before you go live, build a hand-curated evaluation set: 50–100 question-answer pairs where you know what the right answer is and which document it comes from. This takes a day. It will pay for itself every time you change a chunking strategy, swap an embedding model, or tune retrieval parameters.
Without this eval set, every change is a guess. You change the chunk size from 512 to 800 tokens and "it seems better." With the eval set, you run it in ten minutes and know that retrieval precision went from, say, 71% to 79% while recall stayed flat. One is engineering. The other is hope.
RAG systems rarely fail in production because of a weak embedding model. They fail because nobody built an eval set or a freshness pipeline, and the chunking was never designed for the documents people actually query.
Related reading