RAG is not magic: the real cost of giving AI enterprise context
Answer quality is a data and permission problem, not a model problem. What changes when the index must respect who can see what.
*Fifth article in our series on enterprise AI, legacy integration and information governance.*
Executive summary
RAG — feeding the model excerpts from company documents before it answers — is today's default architecture for enterprise assistants. It is also the one that disappoints most when deployed without facing two truths: answer quality is capped by archive quality, and the search index is a copy of your data that can bypass every permission carefully configured in the source systems. This is not a model problem. It is a data and access problem.
How it works, in one business sentence
Before answering, the system retrieves the most relevant excerpts from the company archive and hands them to the model as reference material. Answers then cite internal sources, which reduces invention and raises trust — provided the archive deserves trust.
The three real reasons it fails
1. The archive is wrong. Manuals in three versions, revoked policies, parallel spreadsheets. The system retrieves the document most similar to the question, not the most correct one. If the company does not know which version is current, neither will the AI — and it will state the wrong one with conviction.
2. Permissions were not inherited. When indexing folders and systems, everything often gets flattened into a single index. From then on, a well-phrased question can return HR, legal or M&A content to someone who should not see it. The index must carry source permissions and filter them per user at query time.
3. Nobody owns the archive. Without curation, quality decays: new documents arrive, obsolete ones remain, and user trust falls until the tool is abandoned.
The cost that is not in the proposal
| Item | What it involves | Usually forgotten? |
|---|---|---|
| Document cleanup | Identify current versions, retire obsolete ones | Almost always |
| Permission mapping | Inherit source ACLs and test per profile | Frequently |
| Ingestion and reindexing | Update the index when sources change | Frequently |
| Quality evaluation | A question set with known correct answers | Almost always |
| Inference and storage | Recurring usage and index cost | Rarely |
The recurring operational bill is usually modest next to the effort of putting the document house in order — and that effort creates value even outside AI.
How to tell whether the project is healthy
Three simple indicators, reviewed monthly:
- 1.Share of answers with a valid cited source. An answer without a traceable source does not count.
- 2.Leakage tests per profile. A fixed set of sensitive questions run as users from different departments; any improper result is an incident.
- 3.Index freshness. Average time between a source document changing and the index reflecting it.
What to do on Monday
- Pick a small, well-maintained document domain for the first case — do not index "the whole network drive".
- Name an archive owner with authority to retire obsolete documents.
- Require the vendor to demo permission filtering with two users of different profiles.
- Build a set of 50 questions with known correct answers before go-live; it is the only honest way to compare vendors.
Conclusion
RAG delivers what the archive allows. Treat the project as an information governance initiative — current version, owner, permission, lifecycle — and you get reliable answers. Treat it as plugging in a tool, and you get an expensive search engine with a confident accent. On where to store that archive safely, see The forgotten asset.
Further reading
Engineering track:
- RAG engineering: chunking, embeddings, and reranking — the technical deep dive on this topic.