← Library

Bigger Context Windows Are the Wrong Race

In a blind replay study on our own production conversations in July 2026, a proposed tight context plus retrieval matched a full-transcript window's answer quality at about 15% of the uncached answer-prompt tokens. More context did not reduce hallucination; it slightly increased it. An 80-turn study of why selecting the right context matters more than carrying all of it.

The context-window race rests on an appealing idea: if a model can see more of the conversation, it should remember more, make fewer mistakes, and give better answers. The obvious response to a forgetting assistant is therefore to keep widening the window until the whole transcript fits.

We believed a different architecture should win. Keep a small recent window, maintain a structured account of what is happening, and retrieve older evidence only when the turn calls for it. That should cost less without sacrificing quality. But this is the kind of claim that becomes marketing folklore unless you test it against the strongest alternative, so we replayed real turns and put full context in the comparison as a blind, unlabeled candidate.

The result in one sentence

The proposed tight-context-plus-retrieval configuration matched full-transcript answer quality at about 15% of the uncached answer-prompt tokens in the 80-probe replay. That is not an end-to-end warm-cache claim: in a separate modeled 100-turn chat, the answer prompts used 109,837 cache-adjusted effective input tokens for the proposed configuration and 145,200 for full transcript, while the tight arm also required about 69,000 cheap-model tokens to build its rolling state. Prompt caching and state-building overhead materially narrow, and can erase, the raw-token advantage. The proposed configuration delivered roughly six times more measured answer quality per uncached answer-prompt token than full context.

The other result matters just as much: more context did not make the answers safer. The full-transcript candidate produced slightly more judge-cited inventions than the tight retrieval-fed candidate, including the study's worst single failure: a confident, verbatim-looking provider error and an invented request identifier in the exact conversation where the user was trying to find that real string.

What we tested

The dataset contained 80 real user turns drawn from 14 production conversations: long and short one-to-one chats, multi-party rooms, and a self-chat. Together those conversations contained 2,270 messages. Most probes came from deep enough in the conversation that a short recent window could not simply carry everything forward.

Each turn was answered five ways, with the answering model held fixed:

• Full-transcript oracle: the entire visible transcript up to the test turn, capped only at a very large context limit.
• Vanilla large window: the full transcript until the model's large window required oldest-first dropping.
• ai-advisor compaction: a large transcript using the ai-advisor's specific threshold and compression scheme.
• The current thinqOS configuration: a short recent window, a compact rolling summary and fact ledger, plus limited retrieval.
• The proposed tight configuration: the same short recent window, a richer structured state document, and model-triggered retrieval of older source material across conversation types.

The five answers for every probe were shuffled and shown without labels to a judge. The judge scored context recall, factual consistency, task success, and overall quality, using recent conversation context, the real production reply, and ground-truth excerpts as aids. That produced 400 scored answers. The full-transcript oracle was hidden among them; it received no privileged label. The ai-advisor's compaction trigger fired only once; for about 93% of the conversations that arm behaved like full transcript, so its result is not general evidence for compaction.

Quality was effectively tied

The proposed tight configuration scored 4.92 out of 10 overall. The full-transcript oracle scored 4.78. The vanilla large-window arm scored 4.91, and the compaction arm scored 5.06. Those differences are inside the experiment's noise floor. The honest conclusion is parity, not that one of those four arms was smarter.

The built-in noise check makes that restraint important. On most probes the oracle and vanilla prompts were identical, yet repeated generation and judging still produced a mean absolute per-probe score gap of 1.82. Across 80 probes, mean differences smaller than roughly 0.6 were not interpretable. The study therefore supports “matched quality,” not a claim that tight context beat full context.

The uncomfortable result: today's default was weakest

The quality-matched tight configuration is not the configuration thinqOS ships today. The current configuration scored 4.26, the weakest arm measured, with a paired gap of 0.53 points against the oracle. That gap sat near the experiment's noise boundary overall, but it concentrated in exactly the turns a lossy small context should struggle with: references to old details and multi-party rooms.

The current summary is too small and churns too aggressively. Its fact ledger holds too few specifics. Its retrieval is limited, and some recall logic observes what it would retrieve without actually injecting the source into the answer context. Those are implementation gaps, not evidence that brute force is the right destination.

The proposed arm changed the quality of the small context rather than simply enlarging it: a richer structured state, more durable key facts and open threads, participant state, and model-triggered retrieval of verbatim source material when the turn referred back. That is the direction thinqOS is building toward. Until that work ships and is verified in production, the proposed arm's 15%-of-uncached-answer-prompt-tokens result is evidence for the architecture, not a product specification.

Why the giant window hallucinated more

The judge cited an average of 1.70 inventions per full-transcript oracle answer and 1.51 per proposed tight-context answer. The difference is small, but its direction matters: more raw material did not ground the model more reliably.

A giant transcript gives a small answering model more facts, but also more fragments it can combine into something authoritative-sounding and false. In the worst example, the full-context answer constructed a detailed production-log story around an error string that did not exist. The user was asking for an exact historical detail; the large window provided enough adjacent technical language to fabricate one convincingly.

This does not mean small prompts cannot hallucinate. They can. It means the context window is not a truth mechanism. Relevance, provenance, source selection, and the discipline to say when evidence was not found matter more than raw volume.

Cost, with caching included

“15% of the tokens” compares uncached answer prompts in the July 2026 replay: 3,772 mean input tokens for proposed versus 24,873 for full transcript, a 6.6-times gap. It excludes the work of building rolling state. In a separate modeled 100-turn chat, uncached answer prompts were 446,820 tokens for proposed and 1,210,500 for full transcript, a 2.7-times gap.

Prompt caching narrows the difference, and leaving that out would overstate the economics. In that 100-turn model, permanently warm answer prompts consumed 109,837 cache-adjusted effective input tokens for proposed versus 145,200 for full transcript. The tight arm also paid about 69,000 cheap-model tokens for summarization and state building, so those answer-prompt totals are not an end-to-end savings claim. People also pause beyond cache lifetimes, and adaptive routing changes models while provider caches are model-specific. The honest conclusion is that context selection can reduce prompt volume, but the realized cost depends on cache behavior and the cost of maintaining state. This study did not measure latency directly.

What this study does not prove

• It does not prove complete recall. In a separate July 2026 evaluation of 12 granular identifier lookups whose answers were confirmed in the retained raw messages, the full Mind search pipeline surfaced only 6 of 12. That 50% miss rate is a material retrieval problem, not a rounding error.
• It does not prove the result for every model. The answering model was held fixed at a fast, lower-cost tier. A deep-model replication could find that a larger model extracts more value from a larger transcript.
• It is real but narrow. Fourteen conversations came from one production workspace and were heavy on agent and software-operations work. The 80-probe mean is more trustworthy than the smaller segment results.
• It did not measure latency directly. Input tokens are a useful cost and prompt-latency proxy, not a stopwatch result.
• It does not validate today's default. The proposed arm was a replayed design. Shipping it, measuring it live, and confirming that the runtime behaves like the study arm remain separate work.

The race worth running

The point of a Mind is not to pour everything it knows into every prompt. It is to keep a durable record, decide what governs the task in front of it, and bring the right evidence into a bounded working context with its source intact.

This study strengthens that direction and sharpens the work left to do. Full transcripts were expensive without being better, and more context did not reduce errors. The better target is not the biggest possible window. It is the smallest context that preserves the state of the work and can retrieve the exact old evidence when the moment requires it.

Study run on production conversations and completed in July 2026. Part of the thinqOS science series.