Which does your business chatbot need, RAG or fine-tuning?

For most businesses building a chatbot over their own documents, the answer to RAG vs fine-tuning is to start with RAG. Retrieval augmented generation looks up the relevant passages when a question is asked, so answers stay current, cite their source and respect who may see what. Fine-tuning changes how a model writes or classifies, not what it knows.

Fine-tuning solves a different problem, and some systems need both. Some need neither, because a search box or a form would do the job better. This guide sets out what each approach does, where each one breaks, and how to decide for a business chatbot that has to answer real questions from staff or customers.

What is the difference between RAG and fine-tuning?

Retrieval augmented generation in plain terms

RAG was set out in the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis et al. The authors pointed out that knowledge stored only inside a model's parameters is hard to update and gives no provenance for an answer. Their fix was to pair the model with a searchable index of documents.

In a business system that looks like this. Your documents are split into passages and indexed. When someone asks a question, the system searches the index, picks the most relevant passages and hands them to the language model with an instruction to answer from that material and say where the answer came from. The model itself is unchanged. Update a document and the next answer reflects it.

Fine-tuning an LLM in plain terms

Fine-tuning trains an existing model further on examples of an input and the ideal output, which adjusts the model's weights. OpenAI's supervised fine-tuning documentation accepts as few as 10 examples, suggests starting with around 50 well-crafted ones, and lists classification, specific output formats and fixing instruction-following failures as typical uses. It also advises setting up evaluations before you invest, so you can tell whether the tuned model actually beats the base one.

Microsoft's guidance on fine-tuning is explicit that fine-tuning does not replace retrieval for current information. A useful way to hold the difference: RAG gives the new starter the handbook and lets them look things up; fine-tuning trains them in how you want the work done.

How do RAG and fine-tuning compare?

Two rows matter most for business use. Permissions, because information in a model's weights cannot be hidden from some users and shown to others. And citations, because staff only trust an assistant they can check.

When does RAG win?

RAG is the better fit when one or more of these is true.

• The knowledge changes. Price lists, policies, procedures, product specifications, contract terms. If a document is revised monthly, a fine-tuned model is out of date within a month.
• People need to see the source. Finance, HR, compliance and customer service answers are only useful if someone can click through to the clause or procedure behind them.
• Different users may see different things. HR records, client files, board papers and pricing for specific accounts all need access control per user. The OWASP guidance on vector and embedding weaknesses recommends permission-aware vector stores and strict partitioning of data between user groups for exactly this reason.
• The content is spread across systems. A typical AI chatbot for company documents draws on a document store, a ticketing system, an intranet and a few spreadsheets. RAG can index each source and keep them in sync.

When does fine-tuning help?

Fine-tuning earns its place in narrower situations.

• Consistent tone or format. If every answer must follow a strict structure, such as a JSON payload for another system or a house style for customer replies, examples teach this more reliably than a long prompt.
• Narrow classification. Routing inbound emails into a fixed set of categories, tagging support tickets or extracting fields from a known document type. These are tasks with a right answer you can measure.
• Cost and latency at volume. Microsoft notes that examples generated by a larger model can help a smaller model meet task requirements at lower cost and latency. That matters when you run the same task tens of thousands of times a day.
• Better use of retrieved context. A model can be tuned to rely on the passages it is given and ignore irrelevant ones, which is one reason the two approaches combine well.

Fine-tuning is a poor tool for teaching facts. You cannot easily remove a fact once trained in, you cannot cite it, and you cannot restrict it to certain users. If the main goal is "the bot should know our documents", fine-tuning is the wrong lever.

When do you need both?

You need both when you have current knowledge and a demanding output requirement at the same time. A support assistant that must answer from the latest product manuals and also produce replies in a fixed structure for your ticketing system is a common example.

Microsoft Research's study RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture found the gains stacked: fine-tuning improved accuracy by over 6 percentage points and RAG added a further 5. The order we recommend is RAG first, then fine-tune only where evaluation shows a specific, repeated gap that prompting cannot close. The question logs from a live RAG system are also the best source of training examples you will get.

When do you need neither?

Plenty of chatbot projects should not be chatbot projects.

• People want the document, not an answer. If staff mostly need to find the right form or policy, a good search box with sensible filters and titles will serve them better and cost far less.
• The process is structured. Holiday requests, returns and site inductions are better as forms or guided flows. A form validates input; a chatbot has to guess.
• The answer lives in a database. "What is the stock level for this SKU?" should query your stock system directly, not search exported documents. That is an integration job, not a RAG job.
• There are twenty questions, not two thousand. A short, well-maintained FAQ page beats any model for a small, stable set of answers.

The same thinking applies to back-office automation. In our invoice reconciliation work with a finance team handling around 900 suppliers, the design goal was a 99.98% three-way match rate with only exceptions reaching a person. Narrow, measurable tasks like that reward rules and matching logic first, with a model only where judgement is needed.

What makes a document chatbot work in practice?

The choice of RAG or fine-tuning matters less than the engineering around it. These four areas decide whether a business chatbot is still trusted after the first month.

Chunking

How you split documents determines what the search can find. Splitting by heading and section usually beats splitting by a fixed number of characters, because a procedure step stays with its heading. Keep metadata with every chunk: document title, section, version, owner and date. Tables, scanned PDFs and spreadsheets need their own handling, as plain text extraction often scrambles them.

Permissions-aware retrieval

Filter results by the user's permissions at search time, before anything reaches the model. Do not rely on the prompt to tell the model to withhold information it has already been shown. The simplest robust pattern is to copy the access rules from the source system onto each chunk and apply them as a filter on every query.

Evaluation with real questions

Before launch, collect 100 or more real questions from the people who will use the system, with the correct answer and source for each. Run them against every change you make. Track three things separately: did retrieval find the right passage, did the answer use it correctly, and did the bot say "I don't know" when it should have. Our guide to running an AI proof of concept covers how to set success criteria like these before any build starts.

Keeping sources up to date

Most failures we see after launch are stale or duplicate documents, not model problems. Sync from source systems on a schedule, remove superseded versions, and give each document an owner. When the bot gives a wrong answer, the fix is often to correct the document, which then improves every future answer.

Frequently asked questions

Is RAG cheaper than fine-tuning?

RAG is usually cheaper to build and much cheaper to keep current, because updating knowledge means re-indexing a document rather than retraining a model. Each question costs a little more, since the prompt includes retrieved passages. At very high volumes on a narrow task, a fine-tuned smaller model can be cheaper per call.

Can fine-tuning teach an LLM our company documents?

It can absorb some of the content, but not in a way that suits business use. You cannot cite where an answer came from, update a single fact without retraining, or restrict information to certain users. For questions about company documents, retrieval augmented generation is the better approach.

Does RAG stop a chatbot from hallucinating?

RAG reduces hallucination by grounding answers in retrieved text and showing the source, so errors can be spotted. It does not remove the risk entirely. If retrieval finds the wrong passage, the model can still give a confident wrong answer, which is why testing against real questions matters.

Can we use RAG and fine-tuning together?

Yes. A common pattern is RAG for knowledge and fine-tuning for output format, tone or routing. Research from Microsoft found the accuracy gains from the two approaches were cumulative. Build and measure the RAG system first, then fine-tune only where a specific gap remains.

How long does it take to build a RAG chatbot for company documents?

A focused pilot over one well-organised document set can often be built and tested in weeks. The time goes mostly on cleaning sources, handling permissions and building an evaluation set, not on the model. Projects that span many systems with complex access rules take longer.

Where to start

Pick one group of users and one document set, gather their real questions, and test a RAG prototype against those questions before deciding anything else. If you would like a second opinion on scope, data or whether you need a chatbot at all, our AI chatbot development team is happy to talk it through.