The experiment
We replayed 80 real user turns from 14 conversations in one production workspace, containing 2,270 messages. The same answering model produced five answers per turn. A judge saw the 400 answers shuffled and without strategy labels.
The strategies were a full-transcript oracle, a large window, a particular compaction scheme, the then-current thinqOS configuration and a proposed tight-context-plus-retrieval configuration. This was an internal replay, not a customer outcome study or an independent benchmark.
Quality was effectively tied across four arms
The proposed configuration scored 4.92 out of 10 overall; the full-transcript oracle scored 4.78. The large window scored 4.91 and the compaction arm 5.06. The experiment’s noise check made differences below roughly 0.6 uninterpretable.
The supported conclusion is comparable measured quality in this test, not that the proposed arm was smarter. The compaction trigger fired only once, so its score does not validate compaction in general.
| Configuration | Overall / 10 | Mean input tokens |
|---|---|---|
| Full transcript | 4.78 | 24,873 |
| Large window | 4.91 | 24,873 |
| Compaction; triggered once | 5.06 | 22,137 |
| Then-current thinqOS | 4.26 | 2,260 |
| Proposed tight + retrieval | 4.92 | 3,772 |
The weaker result belongs in the story
The then-current thinqOS configuration scored 4.26, the weakest arm measured. Problems concentrated around older details and multi-party conversations.
The proposed configuration was not the shipping default when the study ran. These results describe the July experiment. They are not a statement about the performance of today’s production configuration.
Read the token result carefully
The proposed arm used 3,772 mean uncached answer-prompt input tokens, compared with 24,873 for full transcript: about 15%. That measure excluded the work of building rolling state.
In a separate modeled 100-turn chat, warm-cache-adjusted answer prompts used 109,837 effective input tokens for the proposed arm and 145,200 for full transcript. The proposed arm also required about 69,000 cheap-model tokens to maintain state. Caching and state-building costs materially reduce, and can erase, the raw-token advantage. No end-to-end savings or latency guarantee follows.
What the study does not prove
The judge cited 1.70 inventions per full-transcript answer and 1.51 per proposed answer. That small observed difference does not establish a general hallucination reduction. In a separate 12-identifier lookup evaluation, the Mind search pipeline found only six answers that existed in retained messages.
One workspace, one answering model tier and a small set of conversations limit generalization. Test other models and your own workload. The result supports evaluating context selection; it does not prove perfect recall, universal superiority or production readiness.