RAG Explained for Business: When Retrieval Isn't Enough

Retrieval-augmented generation, or RAG, is a technique that lets an AI model look up relevant information before it answers, instead of relying only on what it learned in training. It makes answers more specific and current. But retrieval only finds text; it cannot tell which version is true. That is why curated knowledge matters more than the retrieval itself.

What is RAG?

RAG, short for retrieval-augmented generation, is a way of giving an AI model access to information it was not trained on. Before answering, the system searches a collection of documents for passages relevant to the question, then hands those passages to the model along with the question. The model writes its answer from both.

The idea comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, University College London and New York University. They paired a language model with a searchable store of text. According to the paper, the combined system produced answers that were more specific, diverse and factual than the model alone. The authors also pointed to two practical benefits: answers can show where they came from, and the knowledge can be updated without retraining.

Those benefits explain why RAG became the default way to connect AI to business documents. Many products that "chat with your files" use some form of it.

How does RAG work, in plain terms?

It works in three steps. First, your documents are split into passages and indexed so they can be searched by meaning. Then, when someone asks a question, the system finds the most relevant passages. Finally, it gives those passages to the AI model, which writes an answer based on them.

An analogy helps. Imagine a capable new assistant who has never worked at your company. You ask them a question about your refund policy. Instead of guessing, they go to the filing cabinet, pull out the documents that mention refunds, read them and answer. That is RAG.

The quality of the answer depends almost entirely on what is in the filing cabinet. If the cabinet holds the current policy, the answer is good. If it holds three versions from different years, the assistant may blend them.

Where does RAG work well?

RAG works well when the source material is accurate, current and consistent: a well-maintained help centre, a product manual, a policy library with one version of each policy. It is particularly useful for answering questions across large volumes of reliable text that would not fit in a single prompt.

Good fits include customer support answers from a maintained knowledge base, searching technical documentation, and finding passages in contracts or reports. In each case, someone already takes responsibility for the documents being right.

RAG is also useful for scale. A model can only read so much at once, and Anthropic's engineers describe context as "a finite resource with diminishing marginal returns". Retrieval lets a system select the relevant few passages rather than load everything.

Why isn't retrieval enough on its own?

Because retrieval finds text that matches a question, not text that is true. Most business drives hold old proposals, draft pricing, outdated decks and one-off emails alongside the current facts. Retrieval returns whatever matches best, and the AI writes a confident answer from a mix of current and stale information.

Ask a system connected to a shared drive about your pricing. It might retrieve the current price list, last year's proposal with an old figure, and an email offering one customer a discount. All three are real documents that mention pricing. Only one is true, and the AI has no reliable way to know which.

People feel this when AI gets the business wrong. According to Atlassian's 2026 Pulse survey, only 12% of workers say their AI understands their organisation like a tenured senior colleague. Better retrieval does not fix that if the underlying documents disagree.

What should a business do instead of dumping documents into AI?

Curate first, then retrieve. Distil the facts that matter into a small set of canonical files, each statement with a source and an owner, and mark which are approved for public use. Retrieval over that curated set is reliable. Retrieval over the whole drive is a gamble.

A sensible structure has two tiers:

1. Canonical knowledge. Short, sourced, owned files: product facts, pricing rules, objections, the ideal customer, voice. AI reads these first, every time.
2. Reference material. The wider archive of documents, searched when a question needs detail the canonical files do not hold, with answers checked against the canonical facts.

That first tier is what a company brain holds. It does not replace retrieval. Instead, it gives retrieval something trustworthy to find.

Do you need to build RAG yourself?

Usually not. Many AI assistants and business tools now include retrieval over connected files, and connections through standards such as the Model Context Protocol make it easier to plug in your own sources. The work that matters most is not building retrieval; it is deciding what is true and keeping it current.

Building custom RAG makes sense for large, specialised document collections, such as engineering manuals or regulatory archives, where off-the-shelf tools fall short. For most growing businesses, the bigger return comes from curation and ownership. The guide to context engineering covers how to decide what AI should see.

Skein builds the curated tier as the shared knowledge layer of a Company OS, with retrieval over the wider archive where it helps. That suits a business whose documents have grown faster than anyone can keep tidy. A business with a small, well-kept document set may find built-in retrieval is enough.

Frequently asked questions

What is RAG in simple terms?

RAG, or retrieval-augmented generation, lets an AI look things up before it answers. The system searches a collection of documents for relevant passages, gives them to the AI model with the question, and the model writes its answer from them. It makes answers more specific than relying on training alone.

Is RAG the same as a company knowledge base?

No. RAG is a technique for finding and using text. A knowledge base, or company brain, is the content being found. RAG over a curated, current knowledge base works well. RAG over an unsorted drive often returns conflicting versions of the same facts.

Why does RAG give wrong answers?

Usually because the documents it searches disagree or are out of date. Retrieval finds text that matches the question, not text that is true, so old proposals, drafts and one-off emails can end up in the answer alongside current facts. Curating the sources fixes most of this.

Do small businesses need RAG?

Most do not need to build it. Many AI assistants already include retrieval over connected files. The more valuable work for a small business is keeping a small set of accurate, owned facts that every AI reads first, so whatever retrieval is used finds the truth.

What is the difference between RAG and fine-tuning?

RAG gives a model relevant documents at the moment it answers, so knowledge can be updated by changing the documents. Fine-tuning changes the model itself through further training, which is slower and harder to update. For business facts that change often, retrieval over curated knowledge is usually the better fit.

Related reading