AI Context Window Planner — token budget, fit check, overflow strategy across 11 models

Why context windows are budgets, not buffers

Quick answer: the context window is the TOTAL token budget per request — input prompt + system instructions + output generation all draw from the same pool. A model with 200K context and 180K of input has ONLY 20K left for output (minus any reserved system-prompt overhead). Planning for output budget before picking a model is the mistake that drives the "API truncated my response" class of bug. This planner computes input + system + output together and flags fit BEFORE you make the API call.

Different models, different budget geometry. Claude Opus 4.7 has a 200K standard and a 1M context variant (separate tier with different pricing). GPT-4.1 has a 1M variant. Gemini 2.5 Pro is natively 2M — the largest commercial context as of 2026-04. Smaller models like Claude Haiku 4.5 or DeepSeek V3 are 200K / 128K respectively. For a given task, the right model is not "the most capable" — it's the one with the smallest sufficient context at the lowest price meeting your quality bar.

Token estimation — why "words × 1.3" breaks for code and CJK

Quick answer: the English-prose rule of thumb is ~0.75 words per token, or ~4 characters per token. But code tokenizes differently (~3.5 chars/token — more tokens for equivalent-looking characters because BPE splits on punctuation/identifiers); JSON with indentation is ~3.0 chars/token (formatting eats tokens); CJK characters are each approximately one token (so ~1.5 chars/token). If you estimate code-heavy input using prose ratios, you'll be under by 15-30% — enough to silently exceed budget in tight scenarios.

Context windows as of 2026-04

Quick answer: the 2M-context Gemini 2.5 Pro, 1M-context Claude Opus 4.7 1M / GPT-4.1 1M, and 400K GPT-5 are the current premium tier. The 200K tier (Claude Sonnet 4.6, Opus 4.7 standard) covers most tasks at better cost. Below 200K (Haiku 4.5, Llama 3.3 70B, DeepSeek V3, Mistral Large 2 all at 128K) are for lighter tasks or self-hosting considerations.

Context alone is not the selection criterion — output size matters (max output tokens typically 8K-64K depending on model), pricing matters (3-20× spread at the premium tier), and capability matters (reasoning depth, code quality, instruction-following fidelity). The planner surfaces all four per model so the comparison is apples-to-apples rather than "biggest context wins."

Prompt caching — the biggest cost lever for repeated system prompts

Quick answer: if your request includes a large reusable component (long system prompt, extensive tool definitions, context documents that don't change between calls), prompt caching reduces the input cost on that component by 4-10× depending on the provider. Claude caches at ~10× discount ($15/M → $1.50/M on cached); Gemini at ~4×; GPT-4 at ~2.5×. Cache hits typically last 5 minutes from last access (Anthropic) or 60 minutes (Google) per vendor specification.

Use cases where caching wins big: RAG applications with fixed knowledge-base chunks, agentic loops with repeating tool definitions, chat applications with long system prompts that persist across user turns, batch-processing pipelines applying the same prompt template to many inputs. The planner's cache-hit-rate input (step 4) projects cost with cache applied to the specified percentage of input tokens.

Batch API — 50% off for async workloads

Quick answer: Anthropic, OpenAI, and Google all offer "batch API" pricing at approximately 50% of standard per-token cost. Trade-off: batch requests return asynchronously within 24 hours (sometimes within minutes for short queues). For workloads that don't need real-time response — large-scale document processing, historical data enrichment, content-generation pipelines — batch API is a straight 50% cost reduction.

Not a fit for: chat applications, agentic loops, real-time user-facing generation. Is a fit for: nightly data extraction jobs, bulk translation pipelines, periodic content regeneration. Batch API + prompt caching stack multiplicatively (50% × 10× caching = effective 5% of full price on cached portion of a batched request).

Routing — smaller models for simpler queries

Quick answer: not every request needs your premium model. A classifier-style query ("is this message spam?") can often run on a smaller model at 5-20× lower cost with acceptable quality. Typical routing pattern: small model (Haiku / Flash / Mistral Small) handles query-classification + routing decision; premium model handles the queries that actually need depth. Mature deployments route 15-40% of volume to smaller tiers, saving 10-25% of total spend.

Routing adds architectural complexity — you need a classifier, fallback logic when classification is uncertain, and quality monitoring to detect when routing accuracy degrades. Start with aggressive routing (most queries → small; escalate if confidence low), measure quality, tune. The planner's "route-to-smaller" option applies a ~15% volume reduction to the premium-tier projection as a default estimate.

Overflow strategies — what to do when content exceeds budget

Quick answer: if your input doesn't fit, four architectural options in increasing effort order:

1. Route to larger-context model. Simplest fix — if you need 500K of context and your current model is 200K, move to Claude Opus 4.7 1M or Gemini 2.5 Pro 2M. Same API pattern, bigger budget, usually 1.5-3× cost.
2. Summarize then analyze. Hierarchical summarization — chunk content into summaries, then summary of summaries, until it fits. Loses 10-30% fidelity; good for summarization and high-level-analysis tasks.
3. RAG — retrieve top-K relevant chunks. Embed content once; retrieve only the chunks relevant to each query. Typical setup: 1500-token chunks, 150-token overlap, top-K=10-20 per query. Reduces per-call input by 10-100× at the cost of needing embedding infrastructure and acceptance that the retriever may miss relevant context.
4. Agent-loop pruning. For agentic workflows that accumulate history, prune old tool-call results, summarize conversation every N turns, use tool-result truncation. LangChain + LlamaIndex have pattern libraries; OpenAI Assistants + Claude Tools both implement pruning natively.

Truncation (drop input that doesn't fit) is a last resort — silent quality degradation without surfacing the loss to the caller. Only acceptable when content has clear priority ordering (most recent chat turns matter most; oldest can drop).

What this model does not capture

Tokenization rates are averages with ±15% variance. Specific content (a document with lots of numbers, rare Unicode, specific domain jargon) can deviate. For budget-critical applications, run the actual vendor tokenizer on a sample of your real content.

Pricing changes. The 2026-04 values in the tool were accurate at the time of shipping; vendors adjust pricing periodically. Cross-reference the vendor pricing page before committing to a cost projection for production. Vendors also sometimes run promotional pricing for new models — factor that separately.

Capability, not just fit, matters. Claude Haiku 4.5 fits everywhere Claude Opus 4.7 fits, but Haiku may not solve a complex-reasoning task. Use-case quality requirements are the first filter; fit + cost is the second filter among capable candidates. The tool assumes you've already decided what capability tier you need; it helps you choose AMONG candidates at that tier.

Self-hosting isn't cost-modeled. Llama 3.3 70B and DeepSeek V3 are open-weight — self-hosting on your own GPUs changes the cost math entirely (hardware amortization, ops overhead, latency trade-offs). The pricing shown reflects hosted-API cost; self-hosting analysis needs a separate spreadsheet.

Sources and further reading

OpenAI, tiktoken library documentation (github.com/openai/tiktoken) — canonical tokenization rates for GPT models. Anthropic, Claude tokenizer (docs.anthropic.com/en/docs/build-with-claude/token-counting) — official Claude tokenization API + library. Google, Vertex AI count-tokens API (cloud.google.com/vertex-ai/generative-ai/docs/count-tokens) — for Gemini. Anthropic, Prompt Caching documentation — 10× cache discount specification. OpenAI, Batch API documentation — 50% async tier. Chip Huyen, Designing Machine Learning Systems (O'Reilly, 2022) — for general ML system-design patterns. For RAG specifically: Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020) — the foundational RAG paper; Gao et al. Retrieval-Augmented Generation for Large Language Models: A Survey (2024) — current RAG landscape.