A model that can find the right fact in a 128,000-token document is not necessarily a model that uses that fact when forming a judgment. That distinction — between retrieval accuracy and decision influence — is the central finding of a new working paper by Miao Liu (Carroll School of Management, Boston College) and Zhizhe Liu (Columbia Business School, Columbia University).

Why This Result Is Still Preliminary

This paper is described by the authors themselves as a "very preliminary draft. Comments welcome." It was submitted to arXiv on 25 August 2026 (v1), has not been peer reviewed, and has not been independently replicated. Public replication code and data are not yet available; the authors expect to release a replication package around end of September 2026. The findings are inspectable and the experimental design is coherent, but they should be treated as preliminary evidence, not established results.

What the experiment measured

The researchers constructed firm-specific risk disclosures — covenant thresholds, settlement payments, indemnification caps — designed to be coherent with real SEC filing language, for 12 U.S. registrant firms. Each firm received a target version and five neutral replacements; marginal decision influence is the difference between the model's judgment under the target and its average under the five neutral substitutes.

Critically, retrieval and judgment were measured in separate API calls — what a model can state and what moves its recommendation are two independent observations. Context length was varied from 2,000 to 128,000 tokens while focal-firm information stayed fixed. The primary model was Qwen3.5-9B-Base; Llama-3.1-8B and gemma-4-12B provided cross-family replication; qwen3.8-max served as a commercial production reference.

The gap

At 2,000 tokens, Qwen3.5-9B-Base shows a +3.2 percentage-point influence on sell probability (permutation p=0.025). By 8,000–32,000 tokens, that influence is statistically indistinguishable from noise, and it remains at the noise floor through 128,000 tokens. At 128,000 tokens, the same model correctly identifies the target disclosure for 12 of 12 firms with zero false retrievals on neutral controls. It can find the fact. It does not use it.

The pattern holds across model families. gemma-4-12B shows the largest short-context response (+0.047 at 2K) and reaches essentially zero influence at 32K (1/12 firms positive). Llama-3.1-8B declines monotonically from +0.016. The severity-response range — whether the model scales its judgment to how serious a risk is — compresses from 0.23 to 0.04 at 128K, a factor of 5.6, while rank ordering of risks degrades much less. The model can still sort risks; it can no longer size them.

The authors also ran an exploratory replication using 20 actual 10-K disclosures. With the capable Qwen3.5-27B model, the target disclosure produced approximately +0.037 influence in a short (2,000-token) excerpt and essentially no influence in the complete filing (+0.000), while the model correctly retrieved the disclosure in all 20 cases. For an adverse-only subset of 16 disclosures, the short-excerpt effect reached approximately +0.042, with the reported interval excluding zero. The smaller Qwen3.5-9B-Base workhorse did not reliably respond to these real disclosures even at short context — which is why the authors describe the real-filing evidence as capability-conditional rather than universal.

Why it happens — bounded to one architecture

The paper's mechanistic account is performed primarily on Qwen3.5-9B-Base, a pre-RLHF base model the authors describe as "not what anyone deploys." Two transmission channels are identified: a compressed recurrent running summary and an attention-based lookup. Erasing the running summary removes roughly two-thirds of the disclosure's influence; implanting it recreates about half. Disclosure content remains stably encoded at its read position at all context lengths — comprehension is intact; the failure is at transmission, not reading. This detailed causal mechanism is bounded to the Qwen3.5-9B-Base hybrid architecture and should not be assumed to generalize to production deployments.

What workflow architecture changes

In the authors' bounded chunk-and-summarize implementation, the target disclosure disappeared from the consolidated notes in all 24 firm-arrangement cells at 2K — even though the document fit within a single chunk — and in 21 of 24 cells at 128K. This result applies to the authors' tested generic summarization workflow, not to every chunking or summarization architecture.

A targeted, structured extraction placed immediately before the decision stage substantially restored lost influence. In the main experiment on Qwen3.5-9B-Base, this raised context-retention from approximately 12% to 67% of short-context influence, reaching 8.5 percentage points at 128K with all 12 constructed-filing firms moving in the predicted direction.

In a separate workflow comparison across 12 complete real 10-K filings, extract-then-decide produced +0.043 (11/12 firms positive, 95% CI [+0.026, +0.059]) versus −0.004 for direct reading. These are distinct experiments; the 12-case workflow result does not cover all 20 real filings. This is a substantial restoration, not a complete elimination of the gap.

What enterprises should test differently

If you are evaluating an AI analyst, research agent, or document-review system, retrieval accuracy is a necessary condition, not a sufficient one. Evaluation methods that measure only what a model can state may not measure what they claim. The question to add to your validation suite is: when the model retrieves a material fact, does that fact measurably change the downstream recommendation or action in the direction and magnitude it should?

Retrieval should be treated as an intermediate capability. The delegated task is the judgment, not the lookup. A system that passes a retrieval benchmark while failing to integrate retrieved content into decisions has passed the wrong test.

This is not an argument that long-context AI is broken or that all summarization pipelines fail. It is a design and evaluation finding: the architecture of how information moves from retrieval to decision matters, and that architecture is currently invisible to most enterprise validation workflows.

Primary source: arXiv:2608.24842 — Miao Liu and Zhizhe Liu, submitted 25 August 2026.