Retrieval-augmented generation, or RAG, lets a language model answer from your own documents: the system finds relevant passages, hands them to the model and asks it to answer from them. A demo needs a few PDFs and an afternoon. A system staff rely on needs four more things: chunks that follow document structure, permission checks at query time, a test set of real questions and an index that stays current.
Retrieval decides the answer
When a RAG system gives a wrong answer, look at retrieval before the model. If the right passage never reaches the model, no model can answer correctly. Four faults account for most bad answers.
- Chunks cut mid-thought. A fixed split by length can separate a rule from its exception, so the model sees half a clause.
- Tables turned to noise. A table flattened to plain text loses its rows and columns. A price list becomes a string of numbers.
- Old documents outranking current ones. A superseded policy and its replacement look alike to a similarity search, and the old one can rank higher.
- Queries that look the same. Two questions with different meanings can sit close together in embedding space. Without metadata filters, the system treats them as one.
Fix retrieval first. Upgrading the model before that often changes little.
Chunking follows the shape of the document
Split by the shape of the document and not by a fixed number of words.
| Document | Split on | Keep with each chunk |
|---|---|---|
| Contract | Clause and sub-clause | Clause number, parties, effective date |
| Policy or SOP | Numbered section | Version, owner, department, date in force |
| Support article | Question and answer pairs | Product, release, language |
| Spreadsheet or price list | Row groups with headers repeated | Sheet name, period, currency |
| Scanned form | Field groups | Form type, scan quality flag |
A single splitter across all five means at least four are wrong. Metadata does as much work as the text. Fields such as the date a rule takes effect, the kind of document, its owner, its version and its jurisdiction let you filter before you rank: "the current leave policy for the Noida office" can be answered from the one chunk that fits.
Permissions are checked at query time
Check access at the moment a question is asked, for the person asking, under the rules the source system uses. A one-time check at indexing is not enough.
The reason is that a loose index fails quietly. It raises no error. It simply answers from a file the user was never meant to open. Salary letters, board papers and customer records are typical examples. Security reviews tend to turn on this requirement. Index-time checks also go stale: when someone changes team, the index still reflects their old access until the next rebuild.
Where the answers involve personal data, the DPDP Act's duties apply to the system as well. See the DPDP Act and AI systems.
Build the test set before the system
Start with about a hundred real questions from the staff who will ask them, each with a known good answer and the source passage that supports it. Include questions that have no answer in your documents, so the system learns to say it does not know. Method is in building an AI evaluation set.
Measure three things separately.
- Retrieval hit rate: was the right passage among those handed to the model?
- Answer accuracy: was the final answer correct, given what it was shown?
- Citation fidelity: does every cited passage back the claim it sits beside?
Score the stages and not only the output. If the hit rate is low, work on chunking and metadata. If hit rate is high and accuracy is low, work on the prompt and the model. A single blended score shows a fault without showing its location.
Keep the index current
Documents change after launch. Plan the update path before go-live.
- Detect change at the source: a modified date, a version number or a webhook from the document store.
- Re-process only what changed, and replace its old chunks.
- Mark superseded versions with an end date, so retrieval can exclude them.
- Log what was indexed and when, so a wrong answer can be traced to the version it used. See what to capture in an AI audit trail.
- Alert on silent failure. If a source stops updating, someone should hear about it within a day.
When RAG is the wrong tool
RAG answers questions from text. It is a poor fit when the answer is a calculation over structured data, such as a stock balance or a payroll total. For those, query the database directly and let the model describe the result. It also fits badly where one wrong answer carries legal weight and no person reviews it. In that case, return the source passage with the answer and require a reader to confirm.
A build sequence
- Pick one document set with a clear owner, such as HR policies or product manuals.
- Write the test questions with the people who ask them.
- Build ingestion with structure-aware chunks and metadata.
- Add permission checks at query time.
- Score retrieval, then answers, then citations.
- Add the freshness pipeline and the log before opening it to staff.
Questions buyers ask
Why does a RAG system give wrong answers?
Usually because retrieval returned the wrong passage: poor chunking, missing filters, a stale index or look-alike queries.
How should documents be chunked?
By their own structure (clause, section, table), with metadata attached.
How do you enforce permissions?
At query time, against the person asking, with the source system's rules.
How do you know it works?
Run a fixed question set at every change and score retrieval, answer and citation separately.
Where Brihat Infotech fits
Brihat Infotech builds knowledge platforms with ingestion pipelines, evaluation harnesses and freshness checks as part of its custom AI and LLM development work. Start a conversation.