← Back

LLM Context Budgeting: Keep the Evidence That Matters

Build an LLM context budget for instructions, evidence, history, tools, and output. Use deliberate trimming and summaries without losing decisions or source provenance.

LLM context budgeting is a product decision about what information deserves the model's attention on the next turn. The context window is a ceiling, not a target to fill. A request can fit within that ceiling and still bury the one passage needed to answer correctly.

Consider a coding assistant halfway through a migration. Its history includes the user's requirement, a changed file, three failed test outputs, and a long package-manager log. Keeping everything chronologically may crowd out the current code and the exact unresolved failure.

Reserve space before gathering context

Define an input budget that leaves room for the intended output and any provider-specific accounting. Check the actual model's tokenization and limits. Do not estimate capacity from character counts alone when operating near a boundary.

A hypothetical 24,000-token working budget might reserve 3,000 for output, 2,000 for instructions and tool definitions, 4,000 for recent conversation, and the remainder for task evidence plus safety margin. These numbers illustrate allocation; they are not recommendations for a particular model.

The important property is that context assembly knows the budget before it fetches and concatenates text. If it discovers overflow only at the final API call, the fallback is often crude truncation of whatever happened to be last.

Separate soft preferences from hard requirements. The user's current goal and applicable permission constraints should not compete on equal terms with an old diagnostic log.

Rank information by its next use

For each piece of context, ask what decision it helps the model make now. Current source code helps produce a patch. A previous failed hypothesis may prevent repeating a mistake. A complete installation log may contribute nothing beyond its error line and command exit status.

Anthropic's context-engineering guide discusses compaction and structured notes for long-running tasks. It also emphasizes that aggressive summarization can lose details needed later. Use these techniques as controlled transformations, not as an automatic license to discard history.

Keep recoverable references to material removed from the prompt. A file path, document version, or trace identifier lets the agent retrieve details again. A summary saying “tests failed” without naming the command or failure is too vague to support the next action.

Preserve decisions separately from observations

A durable task note should distinguish the user's requirements, confirmed findings, unresolved hypotheses, completed actions, and next checks. Otherwise, a tentative guess can become a settled fact after several rounds of summarization.

An illustrative note might say:

Requirement: preserve existing public URLs.
Confirmed: the router derives slugs from filenames.
Hypothesis: the cache serves an old navigation list.
Completed: reproduced the stale list in a fresh session.
Next check: inspect the response cache headers.

That structure is more useful than a fluent paragraph that merges all five statements. It also makes contradictions easier to spot when new evidence arrives.

Do not convert a document's suggestion into user authorization during compaction. Preserve the source and authority of instructions, especially when the agent can perform external actions.

Pack evidence without burying exceptions

For retrieval-augmented answers, select complementary passages rather than many near-duplicates. Keep headings, source versions, and the exceptions required to interpret a rule. A short passage is not valuable if removing its surrounding condition changes its meaning.

Lost in the Middle reports position sensitivity in the long-context tasks it evaluated. That is a reason to test your context ordering and evidence placement, not a guarantee that one universal ordering will work best for every modern model.

Try controlled variations on your own cases: place the decisive evidence near the beginning, middle, and end; add plausible distractors; and vary the amount of conversation history. Check whether the answer still cites and applies the correct source.

If extra context reduces quality, inspect whether it introduces conflicting versions or distracts from the task. Increasing the model's maximum window does not resolve those information-management problems.

Trim tool output at the source

Tools should return bounded results that match the operation. A search tool can return relevant excerpts and references instead of entire documents. A test runner can preserve full logs in an artifact while returning the failure summary and location.

Be careful with truncation. Cutting the tail of a command output may remove the actual failure, while cutting the beginning may remove the command or environment. Design a task-specific summary and preserve a way to inspect the full artifact.

Record when output was shortened. The model should not assume an excerpt is complete, and operators should be able to explain why a crucial line was absent from context.

For large structured responses, filter fields deterministically before asking a model to summarize. Avoid spending another generation step on removing data your application can identify exactly.

Evaluate the budget as a policy

Use long-running cases with changing requirements, failed attempts, and source updates. Check whether the agent remembers unresolved obligations and avoids repeating completed work. A single-turn answer benchmark does not exercise compaction behavior.

Track input tokens, retrieval volume, trimming events, answer completeness, and repeated tool calls. Reducing token usage is only a win if the task still finishes correctly within acceptable latency and cost.

The RAG chunking guide addresses evidence boundaries before content enters the prompt. The LLM tracing guide helps diagnose where context assembly failed without collecting every sensitive input.

A good context budget makes the next decision easier. It keeps authoritative requirements and relevant evidence close, preserves uncertainty honestly, and leaves the rest retrievable when needed.