LLM context window selection is not a technical footnote — it is a cost and architecture decision that compounds daily. Get it wrong in either direction and you either pay for capacity you cannot effectively use, or you build retrieval infrastructure to paper over a window that is too small for your workload. This brief gives you a framework to make that call with specific numbers, not guesswork.

The Decision

The choice is concrete: given your workload type, query volume, and budget, should you default to a 32K, 128K, or 1M+ token context window when selecting or deploying an LLM?

The models you are most likely evaluating right now: GPT-4 Turbo at 128K tokens, Claude 3 at 200K, Gemini 1.5 Pro at 1M–2M tokens, and — as of late July 2026 — Kimi K3, Moonshot AI's 2.8 trillion parameter Mixture-of-Experts model with a 1 million token context window, now released as open weights under the Kimi K3 License. This decision is specifically about context tier selection. Model quality benchmarks are a separate conversation.

Why This Decision Is Harder Than It Looks

The context window race has moved faster than most teams' deployment assumptions. GPT-4 launched with 8K tokens. GPT-4 Turbo hit 128K in November 2023. Claude 3 followed at 200K, Gemini 1.5 Pro reached 1M–2M tokens in February 2024, and Kimi K3 now delivers a 1M token window on self-hosted infrastructure. The instinct to "just pick the biggest window available" has become an expensive default.

The economics are unforgiving at scale. Filling a 128K context window with GPT-4 Turbo at approximately $0.01 per 1K input tokens costs over $1.28 per query in input alone — before output tokens, retries, or latency overhead. At 1,000 queries per day, that is $1,280 per day purely in input costs. Kimi K3's API pricing is listed at $3 per million input tokens, which is more favorable per token at the 1M tier, but the absolute cost of filling that window — $3.00 per query — still demands serious volume analysis before deployment.

The deeper problem is that raw window size does not translate linearly into usable capacity. Research from Stanford and UC Berkeley documented the "lost in the middle" phenomenon: models exhibit measurably degraded retrieval accuracy for content placed in the middle of long contexts, even when that content sits comfortably within the stated token limit. Empirically, many models effectively utilize only 20–50% of their nominal context window before accuracy degrades. A 128K window does not give you 128K tokens of reliable reasoning — it gives you strong performance at the edges and diminishing returns toward the center.

Option Analysis

Tier	Representative Models	Approx. Cost to Fill	Best Fit	Key Risk
32K	Mistral (RoPE-extended), earlier GPT-4 tiers	~$0.32/query	High-frequency, structured docs	Forces RAG for larger workloads
128K–200K	GPT-4 Turbo, Claude 3	~$1.28–$2.40/query	Mid-frequency analytical tasks	Lost-in-the-middle beyond ~20K tokens
1M+	Gemini 1.5 Pro, Kimi K3	~$3.00–$10+/query	Low-frequency, cross-document reasoning	Latency, cost, utilization uncertainty

32K windows cover roughly 25,000 words — adequate for single-document Q&A, focused code review, short meeting transcripts, or a handful of concatenated reports. Inference cost per query is lowest, and latency is manageable for user-facing applications. The ceiling hits fast: any workflow requiring cross-document synthesis, full contract analysis, or large codebase reasoning will exceed this limit and require chunking or a RAG pipeline. That adds retrieval error surface and engineering maintenance overhead that teams frequently underestimate.

128K–200K windows (GPT-4 Turbo, Claude 3) cover roughly 96,000–150,000 words — a full novel, a substantial codebase, or dozens of reports concatenated. This tier is where most enterprise teams currently operate. The "lost in the middle" degradation becomes measurable at this scale: retrieval accuracy for content positioned beyond roughly the first and last 20K tokens is demonstrably lower. If the critical information in your documents reliably lives at headers, summaries, and conclusions, this tier performs well. If critical data is distributed throughout long documents uniformly, you are paying for capacity that the model will partially ignore.

1M+ windows (Gemini 1.5 Pro, Kimi K3) open up use cases that were architecturally impossible at smaller tiers: full legal discovery sets, entire multi-repository codebases, multi-book corpora, or continuous long-horizon agent memory. Kimi K3 is particularly notable here — its architecture uses Kimi Delta Attention and Attention Residuals specifically designed to improve information flow across long sequences, and its Stable LatentMoE approach activates 16 of 896 routed experts per token, yielding what Moonshot reports as approximately 2.5x improvement in scaling efficiency over Kimi K2. Whether that architectural investment actually closes the "lost in the middle" gap at 1M tokens is a question that benchmark results need to answer for your specific document types before you commit.

Decision Framework

Four criteria determine which tier is right for your workload:

Query frequency. Map expected daily queries × average context fill rate × per-token cost for each tier before selecting. If you consistently run more than 500 queries per day at high context utilization, the 1M tier becomes economically prohibitive on hosted APIs. Self-hosted Kimi K3 eliminates the per-token ceiling entirely for teams with GPU infrastructure, which changes this calculation fundamentally.

Document structure. Does critical information live at known positions — document headers, executive summaries, final clauses? If yes, a smaller window with precision chunking frequently outperforms a larger window with diffuse content, because you sidestep the lost-in-the-middle degradation zone. If your workload requires synthesizing facts scattered across hundreds of pages with no predictable location, the calculus shifts toward the 1M tier.

Latency tolerance. Filling a 1M context window takes time. Real-time applications — copilots, chatbots, interactive code assistants — cannot absorb that latency. Batch analytical pipelines running nightly summarization, compliance audits, or discovery review can. Match the tier to your response time requirements, not your maximum document size.

Build complexity budget. RAG pipelines that substitute for larger context windows require vector databases, embedding pipelines, retrieval tuning, and ongoing maintenance. Teams that lack the capacity to maintain that infrastructure should factor engineering cost into their tier selection — a larger context window that eliminates a RAG system can be cheaper in total cost of ownership even when the per-query token cost is higher.

Recommendation

For most enterprise teams starting today: Default to 128K (GPT-4 Turbo or Claude 3) and instrument your actual context utilization before committing to 1M-tier pricing. Most production workloads do not fill 128K in practice. The ones that do are usually better served by a hybrid 32K + RAG architecture than by brute-force 1M context, because RAG sidesteps the lost-in-the-middle problem by ensuring the model only sees highly relevant content.

For document-heavy analytical workloads in legal, financial, or compliance functions: Evaluate Gemini 1.5 Pro and Kimi K3 directly on your document corpus using retrieval accuracy benchmarks, not general leaderboard scores. The 1M tier is only justified when cross-document reasoning is the core task and query volume stays under approximately 100 per day on hosted infrastructure.

For cost-sensitive or self-hosted deployments: Kimi K3's open-weight release under the Kimi K3 License is the most consequential recent development for teams evaluating long-context infrastructure. At 2.8 trillion total parameters with 104 billion activated per token via MoE routing, it brings frontier-tier 1M context capability to on-premises or private cloud deployments, removing per-token costs entirely for teams with sufficient GPU capacity. Run it against GPT-4 Turbo on your specific retrieval tasks before committing to hosted pricing at scale.

The central takeaway: a well-instrumented 32K + RAG pipeline frequently outperforms a poorly utilized 1M context window on retrieval accuracy while costing a fraction of the price. Bigger windows buy you architectural simplicity and genuine advantage on cross-document synthesis — but only if the model actually uses that capacity reliably. Verify that on your data before you pay for it.

When to Revisit This Decision

Move up a context tier when your average context fill rate exceeds 60% of your current tier consistently over a 30-day window — you are approaching the degradation zone and paying for a ceiling you are about to hit. Revisit your tier downward if instrumentation shows your p95 context fill rate stays below 30% of your current tier; you are paying for capacity that your workload does not use. And if Kimi K3's technical report — expected to follow the weight release — publishes long-context retrieval benchmarks that demonstrate meaningful improvement over the lost-in-the-middle baseline, self-hosted 1M-tier deployments become worth re-evaluating for workloads currently running RAG on 32K.