What you will be able to do
- Explain why per-turn token cost grows with the amount of context an agent carries, and where compaction fits in
- Name the cost and the risk at each end of the compaction dial: aggressive (cheaper, can lose subtle context) and conservative (higher fidelity, more expensive)
- Choose a compaction setting for a scenario based on task length, the cost of a wrong answer, and whether durable facts are stored outside the context
- Measure the tradeoff with cost metrics and quality metrics together, not cost alone
Key concept
The compaction dial — Compaction is a setting you tune, not an on/off switch. Turning it toward aggressive cuts the tokens resent on every turn but risks summarizing away a detail the agent needs later. Turning it toward conservative keeps more of that detail, and you pay for it in tokens and latency on every turn.
1.Why context gets more expensive every turn
To weigh aggressive against conservative compaction, start with the cost driver. Databricks' Genie consumption guide puts it plainly: each turn re-sends the whole chat's context to the model, and an agentic task takes several turns to plan, act and check its work. The guide's own example is that the twentieth message in a chat costs more than the first. Any agent in a reason-act-observe loop behaves the same way. Every tool result, every intermediate plan and every user message stays in the context and is paid for again on each later turn.
The harness has to decide what to do with that growing history. Databricks' AI harness overview describes the job: as conversations get longer, the harness decides what stays active and what gets summarized, and calls this context compaction. Compaction replaces a long run of verbatim history with a shorter summary, so later turns carry fewer tokens. The question for this lesson is how hard to compress. That choice trades token cost directly against how much of the original detail survives.
Checkpoint 1 of 8· Check yourself
A team sees that its agent's later turns cost far more than its early turns, even though the user messages stay short. What best explains this?
The cost grows because the context is resent on every turn and keeps getting larger. This is exactly the cost that compaction is meant to reduce.
“Each turn re-sends the whole chat's context to the model”Source: docs.databricks.com
2.Aggressive compaction: lower cost, risk of losing subtle context
Aggressive compaction summarizes early and hard. It might trigger well before the context window is full, cut many turns down to a few sentences, or keep only the current goal and the latest results. The benefit is immediate and easy to measure. Every later turn carries far fewer tokens, so cost and latency both fall, and the agent stays well clear of the context limit on long tasks.
The risk is harder to see. A summarizer keeps whatever looks important at the moment it runs. Subtle details are the first things to go: a constraint the user mentioned once, an exception to a rule, a file path from a tool output, a reason an approach was rejected. They often look minor until they become relevant twenty turns later. A neighbouring Databricks product shows the same tradeoff. Genie Agents use context filtering to prioritize the tokens they consider most relevant, and the troubleshooting guide warns that quality may be reduced if important context is filtered out. That warning applies to any mechanism that cuts context, and aggressive compaction makes those cuts more often and cuts more each time.
The task goal is usually repeated, central and obviously important, so any summary keeps it. Subtle details such as a one-off constraint, an edge-case exception or the reason an option was ruled out show up once and can look unimportant when the summary is written. Aggressive compaction summarizes more often and keeps less each time, so these low-salience details are the ones most likely to be dropped.
Checkpoint 2 of 8· Check yourself
An architect tunes the harness to compact very early so the agent never gets near a token warning. Token cost drops sharply. What is the main risk they have accepted?
Dropping context lowers cost, but if the dropped context mattered, quality falls. Aggressive compaction makes that more likely because it drops more, and more often.
“quality may be reduced if important context is filtered out”Source: docs.databricks.com
Checkpoint 3 of 8· Exam question
A team builds an agent that drafts legal memoranda over dozens of turns, where subtle qualifications negotiated earlier in the conversation must not be lost. Token cost is a secondary concern compared to accuracy. Which compaction approach should the team adopt, and why?
Correct answer: A — Conservative compaction, because it preserves higher fidelity of prior context even though it costs more tokens
Sources3
3.Conservative compaction: higher fidelity, higher cost
Conservative compaction goes the other way. It compacts late, near the context limit, and keeps more verbatim history when it does. Recent turns might stay word-for-word, and summaries are longer and more literal. Fidelity is higher: when the agent needs an earlier constraint or tool result, there is a better chance it is still in context as originally stated rather than paraphrased.
The cost is the mirror image of the aggressive case. Every token you keep is resent on every later turn, so a conservative setting pays a recurring cost for each piece of retained context, not a one-time cost. Databricks states the general principle in its guidance on multi-turn LLM judges: a larger window also adds token cost and latency. It recommends a bounded window large enough to capture what you are checking for, and not the whole conversation by default. Apply the same idea to compaction. Keeping more history is not automatically safer. It is a purchase, and it only pays off when the extra history actually changes the agent's answers.
Latency. More input tokens per turn means each turn takes longer, and an agentic task runs many turns. A large retained window can also push you closer to the context limit, so when compaction finally runs it has to compress a lot at once. Conservative is not free fidelity. You pay in cost and in speed on every turn.
Checkpoint 4 of 8· Check yourself
A reviewer proposes never compacting and always sending the full history, on the grounds that it is the safest option. Which response best reflects the tradeoff?
Keeping more context costs tokens and time on every turn. The right size is bounded to what the task actually needs, not as large as possible.
“A larger window also adds token cost and latency.”Source: docs.databricks.com
Sources4
4.Choosing where to set the dial
No single setting is right for every agent, so think in terms of what pushes the decision each way.
Lean aggressive when tasks are long and repetitive, when early turns are mostly exploration whose results are superseded, when a wrong answer is cheap to catch and correct, and when token budget or latency is the binding constraint.
Lean conservative when a missed detail is expensive, for example a compliance constraint, a customer's stated exception or a precise identifier, when the task depends on exact wording from earlier turns, and when the conversation is short enough that keeping it costs little.
The most useful lever is reducing what compaction can lose in the first place. Databricks separates session state from memory. As a session accumulates, an agent or your application distills the durable facts out of it and writes them to memory. The session is later cleared or ages out, and the memory persists. If a user's preferences and key decisions are already written to a memory store, a summary of the transcript can be shorter without losing them, which makes aggressive compaction of the working context much safer. In the same way, a session store that persists the full history lets you reconstruct full context on a follow-up message. Compacting what the model sees in its prompt does not have to mean throwing that history away for good.
Checkpoint 5 of 8· Check yourself
An agent must keep its per-turn cost low on long sessions, but it also has to honour user preferences stated early in the conversation. Which design best reconciles the two?
Durable facts written to memory survive no matter what happens to the session, so the working context can be compacted hard without losing them.
“The session is later cleared or ages out; the memory persists.”Source: docs.databricks.com
Checkpoint 6 of 8· Exam question
An engineering team deployed an agent for multi-step data-pipeline debugging with aggressive compaction to minimize cost. After several summarization passes, the agent began repeating tool calls it had already run and contradicting root-cause conclusions it had reached earlier in the same session. What does this outcome illustrate about the compaction tradeoff the team made?
Correct answer: A — The team accepted a level of context-loss risk from aggressive compaction that was too high for a task requiring fine-grained continuity across many steps
5.Measuring the tradeoff, not guessing it
You cannot evaluate a compaction tradeoff from one side. Aggressive compaction always looks good on a cost dashboard, and conservative compaction always looks good in a spot check of one long conversation. To compare settings fairly, run the same evaluation set under each setting and read the cost metrics and quality metrics side by side. Databricks' Agent Evaluation metrics cover both dimensions, so a compaction change shows up as a pair of numbers: what you saved and what it cost you in quality.
| Metric | What it reveals when you change compaction |
|---|---|
| total_token_count | The cost side: how many tokens the more aggressive setting saved |
| latency_seconds | Whether a smaller retained context made the agent noticeably faster |
| correctness | Whether summarizing dropped something the agent needed to answer correctly |
| groundedness | Whether the agent began making up details it no longer had in context |
The pattern to look for is where the curve bends. As compaction gets more aggressive, token count usually falls steadily. Correctness tends to hold for a while and then drop once compaction starts cutting context that matters. The best setting is usually just before that drop. Separate evaluation sets for long sessions matter here, because short conversations rarely trigger compaction and will hide the effect completely.
Checkpoint 7 of 8· Check yourself
A team makes compaction more aggressive and reports a 40% drop in total_token_count as proof the change is safe. What is missing from their evaluation?
Token count measures only the cost side. Quality metrics on the same inputs are needed to show the fidelity side did not get worse.
“Below is a summary of the metrics that Databricks recommends for measuring the quality, cost, and latency of your RAG application.”Source: docs.databricks.com
Checkpoint 8 of 8· Exam question
A retail support agent occasionally needs to recall a customer's order history from many turns earlier, but most turns are routine and low-stakes. The team wants to avoid both the high cost of always preserving full fidelity and the risk of losing the occasional important order detail. Which conclusion best reflects sound tradeoff evaluation for this scenario?
Correct answer: A — Compaction aggressiveness should be tuned between the two extremes so that routine turns are compacted more aggressively while details tied to open orders are preserved with higher fidelity
Sources7
Exam traps
Each one states something that sounds right. Open it to see what is actually true.
1.Compacting as aggressively as possible is always best, because fewer tokens means a cheaper and therefore better agent.Why is that wrong?
Cutting context lowers cost, but if the cut context mattered, response quality falls. Aggressive compaction is a trade, not a free saving.
Covered in Aggressive compaction: lower cost, risk of losing subtle context
2.Keeping the full history (maximally conservative compaction) is the safe default with no real downside.Why is that wrong?
Every retained token is resent on later turns, so a larger window adds token cost and latency. Size the retained context to what the task needs.
Covered in Conservative compaction: higher fidelity, higher cost
3.Automatic compaction clears accumulated context just as well as starting fresh, so there is no need to start a new chat for a new task.Why is that wrong?
Compaction reduces the context but does not remove it. Only a new chat clears it completely, so unrelated work keeps costing tokens until you start one.
Covered in Why context gets more expensive every turn
4.The only way to stop compaction from losing a user's stated preferences is to compact less aggressively.Why is that wrong?
Durable facts can be distilled from the session into memory, where they persist whatever happens to the session. That lets you compact the working context harder without losing them.
Covered in Choosing where to set the dial
Sources
Every claim above is drawn from one of these pages, quoted as it was written on the date shown.
- 1.
“Each turn re-sends the whole chat's context to the model”
↩︎ Why context gets more expensive every turn“Genie Code automatically summarizes, compacts, and truncates long context, but only a new chat clears it completely.”
↩︎ Why context gets more expensive every turn“Genie Code automatically summarizes, compacts, and truncates long context, but only a new chat clears it completely.”
↩︎ Exam trap 3“Consumption grows with both the length of a chat and the amount of context it carries”
↩︎ Prediction - 2.https://www.databricks.com/blog/ai-harnessSecondary source
“the harness decides what stays active and what gets summarized, a process known as context compaction”
↩︎ Why context gets more expensive every turn - 3.
“Genie uses context filtering to prioritize the tokens it considers most relevant to a question.”
↩︎ Aggressive compaction: lower cost, risk of losing subtle context“quality may be reduced if important context is filtered out”
↩︎ Aggressive compaction: lower cost, risk of losing subtle context“quality may be reduced if important context is filtered out”
↩︎ Exam trap 1 - 4.https://docs.databricks.com/aws/en/data-governance/unity-catalog/service-policies/policy-examplesOfficial docs
“A larger window also adds token cost and latency.”
↩︎ Conservative compaction: higher fidelity, higher cost“Prefer a bounded numeric window large enough to capture the pattern you're checking for”
↩︎ Conservative compaction: higher fidelity, higher cost“A larger window also adds token cost and latency.”
↩︎ Exam trap 2 - 5.
“distills the durable facts out of it and writes them to memory”
↩︎ Choosing where to set the dial“The session is later cleared or ages out; the memory persists.”
↩︎ Choosing where to set the dial“distills the durable facts out of it and writes them to memory”
↩︎ Exam trap 4 - 6.
“Reconstruct full context (including tool calls and reasoning) on a follow-up message.”
↩︎ Choosing where to set the dial - 7.https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/evaluate-assess-performanceOfficial docs
“Below is a summary of the metrics that Databricks recommends for measuring the quality, cost, and latency of your RAG application.”
↩︎ Measuring the tradeoff, not guessing it
Also cited
“apply compaction and trimming strategies so that agents can operate efficiently without losing essential information”
↩︎ Key concept