While building the open-source edition of RepoFold, which runs the whole wiki pipeline on local models through Ollama, we hit the most unpleasant class of bug there is: no error, no stack trace, every API call returning success, and output that is just quietly worse than it should be.
The cause is one default. Unless told otherwise, Ollama loads models with a context window of 4096 tokens. Send a longer prompt and it does not reject it. It truncates it to fit and answers anyway.
Why you will not notice
The truncation is invisible from the client side. The HTTP response is a normal 200. The completion has a normal finish_reason. Nothing in the response body says "by the way, I only read half of that."
The only trace is a single line in the server log, which you are probably not watching:
level=WARN msg="truncating input prompt" limit=4096 prompt=7924It gets worse: the truncation keeps the tail of the prompt, not the head. Your carefully written system prompt, the instructions, the output schema: those live at the start, so they are the first thing to go. The model receives an arbitrary chunk of your data with no instructions attached, and then does its best. Its best looks remarkably like a model that has suddenly become stupid.
How it bit us
RepoFold's planning stage sends the model an overview of the repository so it can decide which wiki pages to create. During development, a bug caused a second run to ingest the previously generated wiki output as if it were part of the repository: 51 extra files in the overview, which pushed the planning prompt past 4096 tokens.
The result was not a crash. It was a plan that looked plausible and was quietly thin: fewer pages, whole modules missing, on the exact same repository that had produced a good plan an hour earlier. We spent time suspecting the model, the prompt wording, and the JSON parsing before reading the server log.
That is the signature of this bug: the same setup produces good output on small inputs and inexplicably degraded output on large ones, and every layer in between reports success.
Symptoms checklist
If you run anything against Ollama, suspect truncation when you see:
- Output quality that degrades as input size grows, on the same model and prompt
- A model that suddenly ignores instructions or output formats it handled fine before
- JSON schema violations from a model that produced valid JSON on smaller inputs
- Answers that only reference the last part of what you sent
To confirm, run ollama serve in a foreground terminal and watch for the truncating input prompt warning while your workload runs. A cheap preventive check on the client side: estimate tokens at roughly four characters per token and assert your prompts fit the window you think you have.
The fix
Set the context length when the server starts. Ollama reads OLLAMA_CONTEXT_LENGTH:
# Linux / macOS
OLLAMA_CONTEXT_LENGTH=24576 ollama serve# Windows, current session
$env:OLLAMA_CONTEXT_LENGTH="24576"; ollama serve
# Windows, persistent (restart Ollama afterwards)
setx OLLAMA_CONTEXT_LENGTH 24576You might expect to fix this per request instead. Ollama's native API does accept an options.num_ctx field, but the OpenAI-compatible /v1 endpoint, which is what most tooling and SDKs actually use, has no way to pass it. If your client speaks the OpenAI protocol, the server-level setting is the fix.
One honest caveat: context is not free. The KV cache grows with the window, so a larger context costs VRAM and can push layers off the GPU. On a 16 GB card, 24576 tokens with a 14B quantized model is comfortable; going much higher may not be. Raise it to what your workload needs, not to the maximum the model advertises.
Design around it anyway
We fixed the environment variable and documented it, but the deeper lesson changed the architecture. The local pipeline now makes many small, focused calls of two to six thousand tokens each instead of a few large batched ones, with a hard input budget enforced in code before any prompt is sent. Every call fits comfortably inside even a conservative context window, so a misconfigured server degrades nothing silently.
The general rule, which cost us a debugging afternoon to relearn: on a local inference stack, the absence of an error is not evidence that your prompt arrived intact. Measure at the boundary, in code, every time.
If you want to see the result, repofold generates a documentation wiki from any local repository, fully offline, with every claim citing the exact lines of code. And if your policy allows a cloud service, the hosted version at repofold.dev runs the same pipeline on frontier models, without you having to think about context windows at all.