Put the knowledge in the index, and the model can stay in the building

A precision parts manufacturer asks whether twenty years of maintenance manuals, quotes and complaint records can be turned into something that answers when asked. The second sentence follows at once: none of those files may go to an outside service.

Anyone who builds AI assistants has heard this pair many times. It sounds like two halves of one requirement. It is actually two requirements that fight each other — as long as "knowing the company's material" is taken to mean "the model has to memorise it". Because the models that memorise the most are the largest, and the largest run in someone else's data centre.

This post argues that the conflict is not real. It comes from an assumption that can be put down: that knowledge has to live in the model's weights. Put the knowledge in an index, and let the model read only the few pieces that were chosen, and "knows the company" and "nothing leaves the building" become two consequences of one architecture rather than two goals to be reconciled with budget. The technique is called retrieval-augmented generation, RAG from here on. Every claim about it below carries a source; for the part about our own product, the post says which numbers exist and which do not yet.

Models do not retain the long tail, or last week's change

First, why "let the model memorise it" is a poor plan even when the model is large.

Kandpal and colleagues measured something very direct in 2023: whether a model can answer a factual question is strongly related to how many documents relevant to that fact it saw during pre-training. On "long-tail" facts with few supporting documents, models do markedly worse. The authors' estimate is that closing that gap by scaling alone would take models far larger than today's, and that adding retrieval clearly reduces the dependence on having seen relevant documents in pre-training^[1].

Your company's maintenance manual appears on the internet zero times. It is the long tail of the long tail.

Mallen and colleagues reached the same place from another direction. They built a set of questions about popular and obscure entities, and found models answer well about popular ones and poorly about obscure ones; retrieval helps most on the obscure, and can actually hurt on the popular, which is why they propose retrieving only when needed^[2].

The third thing is time. On FreshQA, Vu and colleagues found that most models do badly on questions whose answers change over time, and that putting search results into the prompt improves them substantially^[3]. Prices, stock, who owns which line: the questions a company asks most are exactly the ones that change.

Three papers, three directions, one conclusion. Weights are good at remembering form and poor at remembering obscure, changing facts. Company knowledge is almost entirely obscure, changing facts.

Separate finding from answering: that is RAG

The name comes from the 2020 paper by Lewis and colleagues. They attached a generative model to a searchable document store: before answering, find the relevant passages, then have the model write from them. The model's weights are "parametric memory"; the document store is "non-parametric memory", and the latter can be opened and inspected, or replaced wholesale — they showed that swapping a 2016 Wikipedia index for a 2018 one updated the model's world knowledge with no retraining^[4].

The "finding" step is a research topic of its own. Karpukhin and colleagues encoded questions and passages as vectors with two encoders and compared distances, beating the classic BM25 keyword search by 9 to 19 percentage points on top-20 passage accuracy in open-domain question answering^[5]. Guu and colleagues went a step earlier and pre-trained the retriever together with the model, so the model learns what to look for, and the documents it finds are ones a person can read^[6]. The field has since grown many variants; the survey by Gao and colleagues sorts them into three generations: pasting retrieved passages into the prompt, adding rewriting and re-ranking around the retrieval, and breaking retrieval into composable modules^[7].

For someone deciding whether to adopt it, though, the variants matter less than the dividing line itself:

A small model reading the right pieces keeps up with a large one

If the model does not have to memorise, how large does it need to be?

RETRO, from Borgeaud and colleagues in 2022, gives a direct number. They let the model retrieve from a two-trillion-token text database while generating, and a model with twenty-five times fewer parameters than GPT-3 matched GPT-3 and Jurassic-1 on the Pile language-modelling benchmark^[8].

Ram and colleagues did a cheaper version in 2023: leave the model untouched and simply prepend documents found by an off-the-shelf BM25 retriever to the input. That alone produced clear gains across models of different sizes^[9]. This paper matters most to anyone adopting the method, because what it shows is that you do not need to change the model, or train a retriever, to get most of the benefit.

Taken together: once "what it knows" has moved into the index, the model's remaining job is to read a few passages and answer well — and an open-weight model that fits in one workstation can already do that.

Send a few pieces, not the whole file

Someone will ask: context windows keep growing, so why not just drop the whole manual in?

Two papers say no. Liu and colleagues found in 2024 that models answer best when the relevant information sits at the beginning or end of a long context, and noticeably worse when it sits in the middle — and that performance falls as the context grows, even for models built for long contexts^[10]. Shi and colleagues measured a different kind of harm in 2023: add irrelevant sentences to arithmetic problems and accuracy drops sharply. The model is led away by what does not matter^[11].

So the "finding" step is not there to save money. It decides whether the model sees three right pieces, or thirty pieces with three right ones hidden inside. It can answer the first. The second is not guaranteed.

Why this pairs so well with a local model

Now the manufacturer's question can be answered. Of the four steps in the path above, only one knows which model it is. Three things follow, and each of them is about running locally.

First, the documents stay in the building. Cutting, scoring and selecting are code, running on your machine. With a local model, no text leaves your environment at any step; even with an outside API, what leaves is those few pieces, never the whole file. The "cannot go outside" requirement is not a constraint in this architecture. It is the default.

Second, prompt length is your own bill. With an outside API, sending a few more pieces costs a little more money. With your own model, it occupies your own GPU and lengthens your own queue. The previous section's two papers say that sending only the right few pieces answers better; this section says it also runs. Two reasons, one practice.

Third, an update is a rebuilt index, not a training run. Ovadia and colleagues compared the two routes directly in 2023: push new knowledge into the model by unsupervised fine-tuning, or connect it by retrieval. Retrieval won consistently, for knowledge the model had seen and for entirely new facts; and they found models struggle to learn new facts through fine-tuning unless the same fact appears in many different phrasings^[12]. On local hardware the difference is practical: fine-tuning needs someone to prepare data, run the training and check nothing else was forgotten. Rebuilding an index is a scheduled job.

We built it as HangRui Index

We built that path on our own site first. This site's assistant cuts the site's pages into a few dozen pieces, scores each against the question, sends only the highest-scoring few to the model, and then matches every citation in the answer back to the sources that were actually sent. Under every answer are three counts — pieces read, pieces sent, citations verified — from the run that just happened. How it is cut and scored, and why there is no framework, and why knowledge belongs in the index rather than the weights, are the two previous posts, in Chinese.

HangRui Index is the same path, pointed at your documents. It cuts what is in your PDFs, Word files, help centre and ticketing system along their own structure and indexes it; the model stage can be an open-weight model in your server room or an outside API, and switching is one line of configuration; every answer carries its sources, every source is verified, and the same counts are printed on the screen. Change a document, rebuild the index, and the next answer is current.

It does not sell a smarter model. It sells the three steps of the path that are not the model, and the fact that where the model stage runs is your decision.

What this costs

If the index is wrong, the model cannot save it. Send the wrong pieces and the model answers confidently from the wrong pieces. This moves the responsibility from "will the model make things up" to "are the documents right, and cut well" — to a place you can fix, but someone still has to fix it.

A local model reads well and writes flatter. The research cited above measures whether answers are right, not whether they are well written. A model that fits in a workstation usually writes less fluently than the largest API models. If the job is "findable and accurate", that is an acceptable trade. If the job is "beautifully written", it may not be.

Someone has to look after the hardware. GPU, drivers, model version, quantisation format: all of them change. An outside API folds those into the bill. Running locally hands them back to you.

The cited work has its scope. The benchmarks in this post are almost all English public datasets: Wikipedia question answering, the Pile, arithmetic word problems. They show that the mechanism holds, not that it will hold on your Chinese maintenance manuals. Each sentence above says what was measured on what; our own number on a local model does not exist yet.

What was left out. How the retriever itself is trained, whether to use vectors rather than keywords, how to evaluate a RAG pipeline: none of that is here. The post makes one argument only, about where the knowledge goes, where the model goes, and why those two decisions are tied together.

Next

This is the third post on that path. The first was about code verifying citations, the second about why knowledge goes in the index, and this one about why the model can stay on your machine.

The next step is to actually connect it to a local model and measure: the same questions, the same index, one stage swapped, and what each of the three counts becomes. When that has been measured it will be written up. Until then, you can ask this site a question and watch those three counts move.