Compaction Prompt Tuning: Recall First, Then Precision

1.What a compaction prompt actually controls

Base models keep nothing beyond their current context window, so a long-running agent depends on its harness to manage memory. When the conversation grows long, the harness decides what stays active and what gets summarized. Databricks calls this context compaction. The summarizer works from an instruction, and that instruction is the compaction prompt. Whatever it fails to ask for is gone once the agent restarts from the summary.

On Databricks, the input to that summary is an agent trace. An MLflow trace records the initial input, the final output, and every intermediate step in between: model calls, tool calls, retrieval, plus each step's inputs, outputs, latency, token usage and cost. A compaction prompt turns that record into something much smaller. Tuning it means deciding which parts of the trace the next stretch of work actually needs.

Sources12

2.Recall and precision as the measuring stick

The objective asks for two passes: get recall high first, then raise precision. Both terms have exact definitions in the Databricks evaluation material. They're written for retrieved chunks, and they carry over directly when the output is a summary instead of a set of chunks.

Recall is about completeness: of everything you know is relevant, how much made it into the output. Databricks describes it as a measure of the completeness of the results. Precision runs the other way: of everything the output contains, how much is actually relevant. The two have different costs. Recall needs a ground truth that lists every relevant item. Precision doesn't, and Databricks notes that an LLM judge can assess the relevance of each item.

That difference in cost explains the order of the passes. A compaction summary that drops a fact can't be repaired later, because the agent no longer sees it. A summary that keeps a few extra lines just costs some tokens. So you first make sure nothing needed is missing, which requires the full ground truth. Only then do you trim.

Sources3

3.Pass 1: maximize recall with expected facts

The recall pass starts with ground truth. MLflow's guide to evaluating prompt versions does this with a dataset of inputs, each paired with the facts that should appear in a good summary. For compaction, each input is the content of an agent trace (or the part about to be compacted). The expected facts are the things the agent must still know after the restart. Traces are the right source for these examples, because Databricks recommends turning representative failures into evaluation and golden datasets.

        "expectations": {
            "expected_facts": [
                "remote work changed collaboration",
                "digital tools adoption",
                "productivity remained stable",
                "challenges with company culture",
                "work-life balance issues",
                "global talent recruitment"
            ]
        }

eval_dataset = mlflow.genai.datasets. ? (
    uc_table_name=EVAL_DATASET_NAME
)

The prompt itself goes into the MLflow Prompt Registry, and each revision becomes a new version. The guide begins with a deliberately bare v1 that tells the model what to do but nothing about what to keep.

prompt_v1 = mlflow.genai.register_prompt(
    name=PROMPT_NAME,
    template="Summarize this text: {{content}}",
    commit_message="v1: Basic summarization prompt"
)

A compaction prompt written like v1 lets the model choose what matters, so recall depends on luck. The recall revision states coverage as an explicit requirement. The guide's v2 includes the line "Include ALL core facts and key findings" along with "Cover all main points mentioned". You then score each version with a scorer that compares the output against the expected facts. In the guide, that scorer is Correctness(), commented as the one that checks expected facts.

scorers = [
    Correctness(),  # Checks expected facts
    sentence_count_judge,  # Custom sentence count judge
]

Make the recall dataset hard on purpose. Databricks advises testing edge cases and including challenging examples in your evaluation dataset. For compaction, that means traces where the important fact is easy to miss: something stated once, early on, and never repeated.

Sources42

4.Pass 2: improve precision without giving back recall

Once the coverage score is where you want it, the second pass removes what doesn't need to be there. The guide's v2 already pairs its coverage rule with a strict output constraint. The constraint is a form of precision pressure, because it leaves no room for filler.

prompt_v2 = mlflow.genai.register_prompt(
    name=PROMPT_NAME,
    template="""You are an expert summarizer. Create a summary of the following content in *exactly* 2 sentences (no more, no less - be very careful about the number of sentences).

Guidelines:
- Include ALL core facts and key findings
- Use clear, concise language
- Maintain factual accuracy
- Cover all main points mentioned
- Write for a general audience
- Use exactly 2 sentences

Content: {{content}}

Summary:""",
    commit_message="v2: Added comprehensive fact coverage with 2-sentence requirement"
)

The MLflow tutorial on evaluating and improving an agent shows the same move in a different prompt. After reading its evaluation results, it adds the guideline "Keep the email CONCISE - only include information directly relevant to the user's request." The compaction version says the same thing about the agent's remaining task: keep only what the next steps need.

The discipline that makes this a precision pass and not a gamble is to evaluate every version against the same dataset. Re-run the expected-facts scorer next to whatever measures the new constraint; the guide pairs Correctness() with a custom sentence-count judge. If tightening the prompt lowers the expected-facts score, you've traded recall for brevity and need to back off. Each change is logged as a new registry version with a commit message that records why it was made, so you can trace a later regression to the edit that caused it.

        eval_results = mlflow.genai.evaluate(
            predict_fn=create_summary_function(PROMPT_NAME, version),
            data=eval_dataset,
            scorers=scorers,
        )

The work doesn't end at deployment. Databricks recommends continuing to evaluate prompts after deployment to catch degradation. For compaction, production traces will keep turning up new kinds of fact the summary drops. Each one becomes a new expected-facts record, which starts another recall pass.

Sources54

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.You can start by making the compaction prompt as terse as possible and add facts back if the agent misbehaves.Why is that wrong?

   Compaction is recall first. A fact the summary drops is gone for the agent, so you secure coverage against expected facts before you add brevity constraints, and you re-check coverage after every tightening.

   Covered in Pass 2: improve precision without giving back recall

2. 2.Precision and recall both require a complete ground-truth list of relevant items.Why is that wrong?

   Only recall needs the complete list. Precision asks whether each produced item is relevant, which can be judged item by item.

   Covered in Recall and precision as the measuring stick

3. 3.A new compaction prompt version can be validated on a fresh set of traces, as long as its score is higher than the previous version's.Why is that wrong?

   Scores are only comparable when every version runs against the same dataset. Otherwise a higher score may just reflect easier traces.

   Covered in Pass 2: improve precision without giving back recall