RAG: making AI answer from your own documents

On 22 May 2020, researchers from Facebook AI Research, University College London and New York University posted the paper that gave RAG its name: retrieval-augmented generation (international · NeurIPS 2020). The idea was to pair a language model with an index of documents, so that it answers from sources and can show where its answer comes from. On 18 July 2024, the French regulator made clear that an organisation connecting an AI system to its own knowledge base is the controller of that processing as soon as the base contains personal data (France · CNIL). In October 2025, Germany’s data protection authorities devoted an entire guidance paper to RAG (Germany · DSK).

The principle fits in one sentence; putting it into practice touches security, data protection and hosting. A RAG system is only as good as its index and its access rights, not just its model.

What to remember

• The principle (international · NeurIPS 2020): instead of relying on the model’s memory, the system first looks for the relevant passages in your documents, then hands them to the model with the question. The model answers from those extracts and can cite its sources.
• The benefit (France · CNIL): answers grounded in data that is “easier to update than the model itself” and more traceable (FAQ of 18 July 2024). For personal data or sensitive documents, the CNIL generally considers on-premise deployment the safer option.
• Access rights (France · ANSSI): they cannot be managed on a model’s training data, but they can be on data added in production, such as a RAG system’s documents (guide of 29 April 2024).
• Erasure (Germany · DSK): entries in a vector database can be deleted one by one, which remains impossible inside the model’s parameters (October 2025).

Going further: how it works

• Chunking. Documents are split into fragments (chunks), usually a few hundred tokens long, tokens being the model’s units of text. Too short, a chunk loses its context; too long, it buries the information.
• Vectorisation. An embedding model turns each chunk into a list of numbers, a vector. Texts with similar meanings produce similar vectors, so the search works on meaning rather than keywords.
• Hybrid search. A classic keyword search (BM25) is added, as it is more reliable on proper names, references and codes. According to a study published on 19 September 2024 (United States · Anthropic), combining the two beats embeddings alone.
• Reranking. A second model rereads the best candidates and reorders them. In the same study, adding a short summary of its context to each chunk cut failed retrievals by 35%, by 49% with hybrid search, and by 67% with reranking on top.
• The context window. The selected chunks go into the prompt sent to the model, which has a limited size. The longer it gets, the less reliably the model finds information in it (United States · Anthropic, September 2025).

The real issue: measuring, and inheriting access rights

A RAG system is assessed on three separate axes. Accuracy: is the answer correct and faithful to the extracts? Coverage: was the right document retrieved? Citations: does each claim point to a passage that genuinely supports it? This requires a reference set of questions, built with the business teams and rerun after every change of model, chunking or index.

The second issue is less visible. A RAG system must apply the source documents’ access rights to every single question. Otherwise, any user can obtain, in one sentence, the summary of a note restricted to senior management. OWASP lists this risk among the top ten for large language model applications (international · OWASP, LLM08:2025). ANSSI recommends reviewing the access rights of generative AI tools as soon as they are switched on, then regularly, for example every month.

Three points to watch

1. Vectors are personal data. Researchers at Cornell University recovered 92% of 32-token texts exactly from their embeddings alone, and retrieved full names from a dataset of clinical notes (United States · Cornell, EMNLP 2023). The vector database must therefore be protected like the documents themselves: encryption, hosting, retention period, and a legal basis, as the DSK points out.

2. Updates and erasure. A document changed or deleted at source must also be changed or deleted in the index. Otherwise, the system cites an outdated version, or data that someone has had erased under the GDPR. This is an advantage of RAG over training, provided the tooling is in place.

3. Costs and alternatives. Every question triggers a search, often a reranking step, then a model call loaded with thousands of tokens. Below 200,000 tokens, about 500 pages, Anthropic considers that the whole knowledge base can simply go into the prompt. Fine-tuning changes the model itself: useful for a style, poorly suited to facts that change, and with no access control over what it has learnt. Finally, some agents search files themselves, with tools such as grep, more slowly than with a prepared index.

What to require from a provider

1. The hosting location of every component: documents, vector database, embedding model, generation model, logs. A so-called sovereign RAG system whose model runs outside Europe is only half sovereign.
2. Access rights inherited from the directory and document management system, checked on every request and tested with several user profiles.
3. No reuse of your data (documents, questions, answers) to train models, written into the processor contract required by Article 28 of the GDPR.
4. An evaluation set delivered with the system, with acceptance thresholds and a rerun after every update.
5. A guaranteed purge time between a deletion at source and its removal from the index, with proof.
6. Logging of questions, retrieved chunks and answers, as ANSSI recommends, with a set retention period.
7. A reversibility clause: export of documents, chunks, metadata and settings.

The model writes the answer; the index decides what it knows, and for whom.

The question to ask on Monday morning

To your CIO and your data protection officer: “If an employee asks our assistant about a document they are not allowed to open, what happens, and can we prove it?”