RAG, short for retrieval-augmented generation, is a way to make an AI model answer from your own documents. Before the model writes a reply, the system searches your files, pulls the few passages that match the question and tells the model to answer from them. A business that wants an AI assistant to know its prices, policies or products almost always needs RAG.
Owners usually ask what is RAG in AI after a chatbot demo goes wrong. The bot quotes last year's prices, invents a return policy or answers a product question with a competitor's spec. RAG is the standard fix, and it is cheaper than most people expect.
On this page
- What is RAG in AI, explained without jargon
- Why a plain chatbot gets your business facts wrong
- How does RAG work step by step
- What makes retrieval accurate or poor
- RAG vs fine-tuning vs long context
- What a RAG system costs to run in 2026
- Where RAG fails and how to test it
- Does your business need RAG
- How RAG connects to AI search and your website
- How to start a RAG project
What is RAG in AI, explained without jargon
Think of the difference between a closed-book and an open-book exam. A language model on its own answers from memory, which is whatever it absorbed during training. With RAG, the model gets to look up the relevant pages first and then write its answer with those pages open in front of it.
Say a 30-person insurance agency in Texas has a 400-page underwriting manual, 60 carrier bulletins and a shared drive of client FAQs. A customer asks whether a detached garage is covered under a homeowner policy. The RAG system searches all of it, finds the three passages that mention detached structures and passes them to the model. The model answers in two sentences and cites the manual section, so a staff member can check it in seconds.
The term comes from a 2020 research paper. Patrick Lewis and colleagues at Facebook AI Research combined a language model with what they called non-parametric memory, "a dense vector index of Wikipedia," searched by a neural retriever. Their RAG models set the state of the art on three open-domain question answering tasks and produced "more specific, diverse and factual language" than the model alone.
Why a plain chatbot gets your business facts wrong
A model like GPT, Claude or Gemini learns from public text up to a training cutoff. It has never seen your price list, your warranty terms or the email your operations manager sent last Tuesday. Asked about them, it does one of two things: it says it does not know, or it writes a plausible answer that is wrong.
The second case is the expensive one. A wrong quote or an invented refund rule reaches a customer with the same confident tone as a correct answer. RAG reduces this by giving the model the actual text to work from and an instruction to stay inside it.
IBM's explainer on RAG is direct about the limit: "While RAG can reduce the risk of hallucinations, it cannot make a model error-proof." The remaining errors have known causes, covered in the testing section below.
RAG also solves a freshness problem. Retraining a model to learn a new price list would cost far more than the price list is worth. With RAG, you update the document, the index refreshes, and the next answer uses the new numbers.
How does RAG work step by step
Every RAG system has two halves. Indexing runs ahead of time, whenever your documents change. Retrieval and generation run each time someone asks a question.
- Collect the sources: help center pages, PDFs, policy documents, product data, past support tickets.
- Split each document into chunks of a few hundred words each, so a search can return the exact passage instead of a 40-page file.
- Convert each chunk into an embedding, a list of numbers that captures its meaning. Chunks about the same topic end up with similar numbers even when they use different words.
- Store the chunks and their embeddings in a vector database, along with metadata such as document date, product line and who is allowed to see it.
- When a question arrives, convert it into an embedding too and find the chunks whose numbers sit closest to it.
- Insert the top few chunks into the prompt with an instruction such as "answer only from the sources below and cite them."
- The model writes the answer, and the application shows the cited sources next to it.
Steps 2 to 5 decide most of the quality. A strong model given the wrong passages will write a fluent, wrong answer. A modest model given the right passages usually gets it right.
What makes retrieval accurate or poor
Anthropic's contextual retrieval study from September 2024 measured the share of searches where the right chunk was missing from the top 20 results, then tested ways to shrink it.
- Adding a short context line to each chunk before embedding, such as which document and section it came from, cut failed retrievals by 35 percent (5.7 percent to 3.7 percent).
- Combining those embeddings with keyword search (BM25) cut failures by 49 percent, to 2.9 percent.
- Adding a reranking step on top cut failures by 67 percent, to 1.9 percent.
Keyword search matters because embeddings are weak on exact strings. A part number such as HX-2207 or a policy code means little to an embedding model, while a keyword index finds it at once. Most production systems now run both and merge the results.
More context is not always better either. The Lost in the Middle study by Nelson Liu and colleagues, published in TACL in 2023, found that models perform best when the relevant information sits at the beginning or end of the input, and noticeably worse when it is buried in the middle of a long context. Five well-chosen chunks beat 50 loosely related ones.
RAG vs fine-tuning vs long context
Three methods get your information into an AI system, and each fits a different problem. The table compares them on the points that decide the choice.
Factor | RAG | Fine-tuning | Whole knowledge base in the prompt |
|---|---|---|---|
What it changes | What the model can look up | How the model behaves and writes | What the model reads on every call |
Best for | Facts that change: prices, policies, catalogs | Fixed format, tone or classification labels | Small, stable sets under about 200,000 tokens |
Updating | Edit the document, re-index in minutes | Retrain on new examples | Edit the text block |
Shows sources | Yes, per answer | No | Possible, less precise |
Main weakness | Bad retrieval means bad answers | Weak at recalling specific facts | Cost grows with every question |
OpenAI's guide to LLM accuracy draws the same line. It treats missing knowledge as a context problem solved with RAG, and inconsistent formatting as a "learned memory" problem solved by showing the model examples, which is what fine-tuning does. It also says "prompt engineering is typically the best place to start."
Anthropic's contextual retrieval article adds a shortcut. For knowledge bases under 200,000 tokens, roughly 150,000 words, you can put the entire knowledge base in the prompt and skip retrieval. The recommendation here is simple: under that size and with steady traffic, try the long prompt with caching first. Above it, use RAG, and keep fine-tuning for tone and format.
What a RAG system costs to run in 2026
As of October 2026, OpenAI lists its text-embedding-3-small model at $0.02 per million tokens. Its API pricing page charges $0.10 per GB per day for file search storage after 1 GB free, plus $2.50 per 1,000 search calls. Anthropic's pricing lists Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output tokens, with cached input read at 5 percent of that rate.
The example assumes 2,000 documents of about 1,500 words each, or roughly 4 million tokens, and 3,000 questions a month. Each RAG answer sends about 3,100 input tokens (five chunks plus instructions) and returns 300.
Cost item | How it is calculated | Monthly cost |
|---|---|---|
One-time embedding of 4 million tokens | 4 x $0.02 | $0.08, once |
Answers on Sonnet 5.5 | 3,000 x $0.0092 per answer | $27.60 |
Managed file search calls, if used | 3,000 x $2.50 per 1,000 | $7.50 |
Managed storage for a 3 GB store | 2 billable GB x $0.10 x 30 days | $6 |
Long prompt instead, 150,000-token handbook, no caching | 3,000 x $0.30 input | $900 |
Same long prompt with cache reads | 3,000 x $0.015 input | $45 |
The token bill for a busy small-business assistant lands well under $100 a month. The real spend is building the pipeline, cleaning documents, setting permissions and testing. The guide to what an AI chatbot costs to build breaks those costs down.
Where RAG fails and how to test it
RAG lowers error rates without reaching zero, and the best public evidence comes from law. A 2024 study by Varun Magesh, Daniel E. Ho and colleagues tested RAG-based legal research tools from LexisNexis and Thomson Reuters, whose marketing had claimed answers free of hallucinations. Each tool hallucinated between 17 and 33 percent of the time, and the authors called the vendor claims "overstated."
Most failures trace back to a handful of causes:
- Stale or conflicting documents, such as two versions of a price sheet in the same folder.
- The right answer was never written down, so the model fills the gap.
- Retrieval returned a similar but wrong passage, for example the 2025 policy instead of the 2026 one.
- Permission leaks, where an employee-only document answers a customer question.
- Instructions hidden inside a document or web page that the model follows, known as prompt injection.
Testing catches these before customers do. Write 50 to 100 real questions with known correct answers, drawn from your inbox or support tickets. For each one, check two things separately: did retrieval return the right passage, and did the answer stay faithful to it. Re-run the set after every document update, and require the bot to say "I don't have that information" when retrieval finds nothing relevant.
Does your business need RAG
The right choice depends on where your information lives and how fast it changes. The table maps common situations to a recommendation.
Your situation | Recommendation |
|---|---|
Under 50 pages of FAQs that rarely change | Put the text in the prompt and skip RAG |
Hundreds of documents, updated monthly or faster | RAG with keyword plus embedding search |
Live data such as orders, stock levels or bookings | API or database lookups as tools, not RAG |
Need a fixed tone or output format | Prompt examples first, fine-tuning if that fails |
Mix of documents and live records | RAG for documents plus tool calls for records |
Regulated advice in health, legal or finance | RAG with citations and human review before sending |
Live records deserve a note. Order status or inventory changes by the minute, so it belongs in a direct lookup, which is the job of the tools described in the guide to how AI agents work. Many agents use RAG as one of their tools, searching documents for the policy and calling the order system for the facts.
How RAG connects to AI search and your website
The same pattern runs the AI answers your customers already use. When ChatGPT searches the web or Google shows an AI Overview, the system retrieves pages first and writes an answer from them. That is why clear, well-structured pages get cited, a topic covered in the guides to getting recommended by ChatGPT and generative engine optimization.
Content that works for your own RAG system works for public AI search too. Short sections with one answer each, dated policy pages, plain headings and one current version of every document help both retrievers find the right passage.
How to start a RAG project
A focused first project takes one use case from idea to measured results. These steps keep the scope tight:
- Pick one audience and one job, for example customer questions about shipping and returns.
- Export 100 real questions from your inbox, chat logs or tickets.
- Gather the documents that answer them, delete outdated versions and fix gaps where no document answers a common question.
- Tag each document with a date, an owner and who may see it.
- Build a prototype on a managed option such as OpenAI file search or Azure AI Search, or on a framework like LangChain.
- Test against your 100 questions and record retrieval hits and answer accuracy.
- Launch with visible citations and a clear handoff to a person.
- Assign an owner to update documents and re-run the test set each month.
Start with step 2 this week, because the question list tells you whether your documents can answer what customers ask. The AI development team at Techno Mantra builds RAG assistants on OpenAI, Claude or Gemini, inside websites, apps and CRMs, with permissions and test sets included. You own all code, data and accounts. Send your use case through the contact page or to info@technomantraa.com, and every inquiry gets a reply within one business day.
Frequently asked questions
RAG stands for retrieval-augmented generation. The term comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, which paired a language model with a searchable index of documents so the model could look up facts before generating an answer.
ChatGPT uses the RAG pattern when it searches the web or reads files you upload: it retrieves text first and answers from it. Without those features, it answers from its training data alone. A business RAG system applies the same idea to your private documents.
No. RAG lowers the hallucination rate but does not remove it. A 2024 study of RAG-based legal research tools found hallucinations in 17 to 33 percent of answers. Citations, a tested question set and an instruction to say when information is missing keep the remaining errors visible.
A vector database stores text chunks alongside embeddings, lists of numbers that capture meaning, and finds the chunks closest to a question. Options range from managed services such as Pinecone to the pgvector extension for PostgreSQL, which suits teams that already run PostgreSQL for other data.
RAG works best on text such as policies, manuals and articles. Spreadsheets and databases with live numbers, like inventory or order status, are better served by direct queries that the AI calls as a tool. Many business assistants combine both: RAG for documents and database lookups for records.