Free tokens change the design: porting a cost-optimized LLM pipeline to local models

When we open-sourced the RepoFold pipeline so it could run offline on local models, the naive plan was to swap the model client and ship. Point it at Ollama, keep everything else, done.

The first wiki it produced was bad. Not broken: bad. Pages of confident, vague prose, whole modules described in two sentences, and citations that pointed at line ranges which did not exist. The same pipeline that produces solid documentation on frontier models produced something closer to a marketing summary on a 14B model running on a single consumer GPU.

The instinct is to blame the model. The actual problem was that we had ported an optimization along with the code, and that optimization was now pointing the wrong way.

The optimization you inherit without noticing

In the hosted product, every token is money. Indexing a repository costs real dollars per commit, so the pipeline is built to minimize call count and maximize what each call does. It batches: forty file summaries in one request, a whole page in one generation, a 48k-token context packed with as much material as fits. Fewer, larger calls is simply the cheaper shape, and on a frontier model with a large context and strong long-range attention, it works well.

Now run that shape locally. Tokens are free. Nothing costs money; only wall-clock time and VRAM. And the model you are running has a fraction of the parameters and, in practice, a much smaller usable context.

So the batching that saved money in the cloud is now pure downside. You are handing a small model the single task it is worst at, digesting a large heterogeneous blob and synthesizing across all of it, and you are getting nothing back for it, because the thing it was optimizing for no longer has a price.

The fix is not a better prompt. It is inverting the optimization.

Many small calls instead of few large ones

The local edition runs what we call deep mode, and its rule is the opposite of the cloud's: make as many small, narrowly scoped calls as the task allows, and cache all of them aggressively.

• One call per file, not forty. Every important file gets analyzed on its own with its full source in context, producing a structured summary: purpose, key symbols, main flows, gotchas, configuration keys.
• One call per symbol. For each selected function, class, or type, a call receives the exact source slice for that symbol plus its importers, and returns two to five factual claims about it. Nothing else is in the context to get confused by.
• One call per section, not per page. Pages are outlined first, then each section is written separately, with an explicit list of what the other sections cover so it does not repeat them.

Every one of these fits in two to six thousand tokens. That matters twice: small models are far more reliable on small grounded tasks than on big synthesis, and small prompts sidestep the silent truncation that a default Ollama context inflicts on longer requests.

The cost is call count. A repository that took a handful of calls in the cloud takes around a hundred locally. On a local GPU that is a fine trade, because the resource you are spending is the one that is free.

Let the harness write the citations

Splitting the work fixed the vagueness. It did not fix the citations, because line numbers are the one thing you should never ask a small model to produce. It will copy worker/pipeline.ts:42-88 as :42-89, or invent a range that looks reasonable, or cite a file that was only mentioned in a summary.

So in deep mode the model never writes a citation. Every mined fact gets an opaque six-character ID, and the model is asked only to end its sentences with markers like [[f:3a9c1f]]. After generation the assembler substitutes the real file and line range, taken from parser data, because tree-sitter knows exactly where every symbol starts and ends. Markers that do not resolve to a known fact are stripped.

The result is not "fewer invalid citations". It is that an invalid citation cannot be constructed. The model is not in the citation-writing business at all.

There is a second benefit that only shows up on the second run. Because a fact's identity is derived from its content rather than its position, code that merely moves does not invalidate anything: the fact is re-anchored to its new lines and the page is reassembled without a single model call.

What it measures

Acceptance run on vercel/ms with qwen3:14b, single RTX 5070 Ti, deep mode:

The incremental behaviour matters more for daily use. Re-running with no changes costs one second and zero model calls. A one-line edit to a single function cost one file summary, one re-mined symbol, twenty-seven facts carried forward untouched, and regeneration of only the sections that cite that function.

Deep mode is the default in the open-source CLI. The old batched path is still there behind --fast if you want the quick version.

The general lesson

Every pipeline carries optimizations shaped by the environment it grew up in, and those assumptions are usually invisible because they were never written down as decisions. Ours was "calls are expensive, so batch aggressively", which is correct in the cloud and actively harmful on a local GPU where the same batching just hands a small model a task it cannot do.

When you port an LLM pipeline to a different environment, the useful question is not "will the model keep up". It is "which constraint was this design shaped around, and is that constraint still real". Here it was not, and inverting one assumption did more for output quality than any amount of prompt engineering would have.

The local edition is at github.com/Neilbenji/repofold, AGPL-3.0, and runs entirely offline on your own hardware:

ollama pull qwen3:14b
repofold generate path/to/your/repo --model qwen3:14b --serve

The hosted product at repofold.dev runs the same pipeline on frontier models, where it writes better prose and adds automatic sync, search, Ask, and an MCP server for your coding agents. Which one is right depends entirely on whether your code is allowed to leave the building.