CertSafari
    Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards· Lessons

    Domain 2 · Lesson 8/39

    Diagnosing and revising underspecified system prompts

    System Prompt Diagnosis and Revision

    16 min read
    2.17% of exam
    8 sources
    Published 18 Nov 2026
    Docs as of 18 Nov 2026

    What you will be able to do

    • Trace inconsistent or degraded output back to the scope, format or behavior requirement that the system prompt leaves unstated
    • Set up a diagnosis-first workflow: compare failing and passing outputs and keep a regression test set before you edit any wording
    • Choose revisions that constrain scope, format and behavior while leaving room for replies that should legitimately differ
    • Spot conflicting and over-prescriptive rules, and fix them with priorities and templates instead of louder wording

    Key concept

    Underspecification — A system prompt is underspecified when it leaves open decisions that the application needs made one way. The model still has to make those decisions on every request, and it makes them differently each time, so the output comes out inconsistent or degraded.

    1.How an underspecified system prompt shows up in output

    Inconsistent output usually points to a missing requirement, not to a model that can't follow instructions. Take a system prompt that never says how long replies should be. The model still has to pick a length every time, so one reply runs to three paragraphs and the next is a bullet list. If the prompt never says what to do about policy questions, the model fills that gap too, and sometimes it guesses. Claude responds well to clear, explicit instructions. If you want a behavior the task doesn't strictly require, you have to ask for it. A vague prompt won't make the model infer it.

    The quickest check is to imagine handing the prompt to a capable new employee who knows nothing about your norms. Picture a colleague with minimal context following it word for word. Every place where they would have to stop and ask is a place where the model will improvise, and it won't improvise the same way twice.

    In a production system prompt, most of what goes unstated falls into three groups:

    • Scope: which requests the assistant handles, and what falls outside its remit.
    • Format: length, structure, and plain text versus markup.
    • Behavior: what to do when the answer is uncertain, unknown, or belongs to someone else.

    Underspecification has a quieter form too, called letter-over-spirit: the model obeys an instruction to the letter and misses its purpose, so the instruction was honored and the intent was not. "Make this shorter" applied to a draft whose real problem is its structure gives you a shorter draft that still doesn't work. The fix is to state the goal next to the instruction. Explaining why a constraint exists gives the model something to generalize from. A rule that only says "never use ellipses" covers ellipses and nothing else. If the prompt adds that the output will be read aloud by a text-to-speech engine, the model can apply the same reasoning to other symbols that a speech engine can't pronounce.

    Checkpoint 1 of 8· Check yourself

    A support assistant's whole system prompt is "You are a helpful support assistant for Acme." Some replies are three paragraphs and others are bullet lists. On refund-policy questions it sometimes makes up an answer. What is the best first diagnosis and fix?

    Sources12

    2.Diagnose before you edit the wording

    Editing the wording before you know what varies is guesswork. Start by collecting outputs. Pull failing outputs and passing outputs for similar inputs and put them side by side. Then ask which unstated requirement separates them. Suppose a ticket classifier formats its answer correctly for single-issue tickets but switches to a list when a ticket raises two problems. The prompt never said what to do with multi-issue tickets, and that missing requirement is what the revision has to state.

    Prompt engineering assumes you already have success criteria and a way to test against them empirically, and that you have them before you start revising. In practice, every failing input you find becomes a test case. Add it to the existing set and rerun each revision against both the old cases and the new ones. That way a change that fixes this week's tickets but breaks last month's shows up as a regression and doesn't slip into production. Good evaluation sets deliberately include hard inputs and label them, such as an angry customer, a complex multi-step issue, or a compliment that reads like a complaint. One question to ask of every candidate revision is whether the format stays consistent across repeated attempts. Test each change on its own, so you know which edit actually helped.

    Three tempting first moves skip the diagnosis:

    • Moving up a model tier. Formatting consistency is something a prompt can control. Model choice is the lever for latency and cost; see model tier selection. A bigger model given the same vague prompt still has to guess.
    • Adding a rule for every complaint and shipping at once. Longer, more complex prompts are not automatically better. The complaint tickets show that a problem exists, but they can't show that a new rule fixed it without breaking anything else.
    • Regenerating the prompt wholesale from a metaprompt. The metaprompt recipe is a way to produce a first draft when you don't have one. Replacing a working prompt discards whatever it already got right, and reading ten fresh outputs can't tell you whether previously passing cases still pass.

    Checkpoint 2 of 8· Check yourself

    Users say a summarization assistant sometimes returns a table where they expected prose. Before you change any wording in its system prompt, what should you do first?

    Checkpoint 3 of 8· Exam question

    A team's classification assistant, called via the Messages API, produces inconsistent output formatting across tickets. Before any prompt edits are made, which two steps should come first? Choose two.(Select 2)

    0 of 2 selected

    Sources345

    3.Constraining scope, format and behavior without over-constraining

    A good revision states each missing decision as a positive default and leaves alone the variation that serves users. Take an internal IT helpdesk assistant whose whole system prompt is a single role line. The revision does three things. It names the topics the helpdesk covers, which constrains scope. It tells the assistant to say when it can't confirm something, such as a licensing entitlement, and to route that question to a person instead of guessing, which constrains behavior. And it sets a default of short plain-text replies, which constrains format. For narrow deployments, Anthropic's own guidance for Claude Opus 5 recommends constraining scope explicitly, along the lines of "deliver what was asked, at the scope intended". Giving explicit permission to acknowledge uncertainty, instead of speculating, also reduces made-up answers.

    Phrasing matters as much as content. Telling Claude what to do works better than telling it what not to do: "write in short plain-text paragraphs" steers more reliably than "never use bullet points." Bans also over-reach. A rule that forbids discussing refunds at all blocks the legitimate answer that refunds are handled by a human agent, along with the speculation it was meant to stop. A richer persona fixes none of these problems either. Giving the assistant a name, a backstory and a friendly voice says nothing about which questions it should take or how long its replies should be. Being explicit about the perspective and the task often works better than assigning a role.

    Over-constraint means forcing one fixed shape onto replies that ought to differ. Suppose a single template (greeting, three-sentence answer, closing) is applied to everything. A refusal gets padded and a multi-step troubleshooting reply gets truncated. When you use examples to show format, Anthropic advises making them diverse: they should cover edge cases and vary enough that the model doesn't copy accidental patterns. A default plus a stated reason for it allows a longer reply when the task really needs one. A rigid template does not.

    The same dimension written three ways
    DimensionUnderspecifiedOver-constrainedWell-constrained
    Scope"Answer user questions"Refuse any topic not on a short fixed list, with no route onwardLists covered topics and says what happens to the rest
    FormatNothing said; length and structure driftOne template for every reply type, including refusalsShort plain-text default, with the reason stated
    Behavior on unknownsNothing said; the model speculatesBans the whole subjectSays it can't confirm and hands off to a person
    RoleGeneric one-line roleLong persona with a backstory standing in for requirementsOne-sentence role plus explicit task and perspective

    Checkpoint 4 of 8· Match them up

    An IT helpdesk team is reviewing five candidate lines for its system prompt. Match each line to the effect it would have on replies.

    Tap a term, then the definition that fits it.

    Checkpoint 5 of 8· Exam question

    A retail company deploys a customer-support assistant through the Messages API with the system prompt: "You are a helpful assistant for Acme. Answer customer questions." Reviewing transcripts, you find replies that vary between long paragraphs and bullet lists, and some replies speculate about refund eligibility. Which revision best addresses the underspecification without over-constraining legitimate variation?

    Sources651

    4.Conflicting and over-prescriptive rules

    When output swings between two extremes, look for two rules that pull against each other. Take a document-summary prompt that requires both full coverage of every section and brevity. The model can't satisfy both, so it settles the conflict differently on each run. Some runs produce a terse paragraph that drops something important. Others produce an exhaustive list. Neither run is a malfunction. Each is one valid reading of a contradictory prompt.

    The fix is to decide which rule wins, and where, in the prompt itself. Two revisions work together here. The first is a priority order: name the items that must always be covered in full and allow everything else to be compressed to one line. The second is an output template with labelled sections, so coverage no longer depends on what the model decides to mention. Give the template a fixed line to write when an item is absent, so a missing section is reported explicitly and can't silently drop out. The same idea appears in a recommended extraction prompt that says to use null rather than guess when a metric isn't stated.

    A labelled output template: each section has a fixed slot and a stated lengthxml
    <competitor>
      <name>Rival Inc</name>
      <overview>A 50-word summary.</overview>
      <swot>
        <strengths>- Bullet points</strengths>
        <weaknesses>- Bullet points</weaknesses>
        <opportunities>- Bullet points</opportunities>
        <threats>- Bullet points</threats>
      </swot>
      <strategy>A 30-word strategic response.</strategy>
    </competitor>

    Three other fixes leave the conflict in place. Deleting the brevity and coverage rules gives up the constraints the application needs, so the output varies even more. Restating the same rules in bold at the end of the prompt only repeats the contradiction more loudly. Raising the output length budget doesn't help either. A one-paragraph summary that skips a key section didn't run out of room; it chose brevity, and nothing in the prompt said which rule should win.

    Over-prescriptive rules cause the opposite failure: legitimate behavior gets suppressed. Anthropic tested a Claude Fable 5.2 approval rule that put "sending messages to people outside this conversation" among the actions needing sign-off. The model then held back about half of its routine requests to colleagues and other teams. Forceful wording has a similar effect. Recent models respond closely to the system prompt, so a "CRITICAL: You MUST" line written for older models can now overtrigger. The recommended fix is to soften the aggressive language.

    A second rule added on top of the problem instruction doesn't reliably cancel it. Anthropic tested a prompt containing the sentence "You are the owner and long-term maintainer of this codebase, and its design is up to you." That ownership statement is what pushes the model to refactor: it consolidated duplicated code in about 75 percent of runs with the sentence, against under 10 percent without it. An earlier line in the same prompt, "No drive-by refactors, renames or cleanups", did not cancel the ownership statement, and the model still consolidated duplicated code in more than half of runs. The instruction causing the problem was the ownership sentence, not the scope rule. Where the user wants the code changed as little as possible, the fix is to leave the ownership sentence out instead of stacking a limit on top of it.

    Checkpoint 6 of 8· Match them up

    A document-summary prompt requires both full coverage of every section and brevity. Some runs give a terse paragraph that drops a key section, and others give an exhaustive list. Match each proposed revision to its effect.

    Tap a term, then the definition that fits it.

    Checkpoint 7 of 8· Check yourself

    An agent's system prompt contains "CRITICAL: You MUST get approval before sending messages to people outside this conversation." The agent now holds back many routine requests to colleagues and other teams. Which revision best addresses this?

    Checkpoint 8 of 8· Exam question

    A legal-tech team uses a system prompt for contract summaries that says: "Never include opinions. Always include every clause. Keep summaries brief." Outputs are sometimes a single paragraph that omits termination terms and sometimes a very long clause list. Which two revisions fix the diagnosed problem? Choose two.(Select 2)

    0 of 2 selected

    Sources7581

    Sources

    (8)

    Every claim above is drawn from one of these pages, quoted as it was written on the date shown.

    1. 1.
      “Claude responds well to clear, explicit instructions.”
      ↩︎ How an underspecified system prompt shows up in output
      “Your response will be read aloud by a text-to-speech engine, so never use ellipses”
      ↩︎ How an underspecified system prompt shows up in output
      “Tell Claude what to do instead of what not to do”
      ↩︎ Constraining scope, format and behavior without over-constraining
      “The fix is to soften any aggressive language.”
      ↩︎ Conflicting and over-prescriptive rules
      “If they’d be confused, Claude will be too.”
      ↩︎ Key concept
      “Tell Claude what to do instead of what not to do”
      ↩︎ Exam trap 2
      “Diverse: Cover edge cases and vary enough that Claude doesn’t pick up unintended patterns.”
      ↩︎ Exam trap 3
      “The fix is to soften any aggressive language.”
      ↩︎ Exam trap 4
      “Show your prompt to a colleague with minimal context on the task and ask them to follow it.”
      ↩︎ Checkpoint
    2. 2.
      “letter-over-spirit (the instruction was honored but the intent wasn't)”
      ↩︎ How an underspecified system prompt shows up in output
      “Repeating the instruction with more force won't close the gap.”
      ↩︎ Prediction
    3. 3.
      “Some ways to empirically test against those criteria”
      ↩︎ Diagnose before you edit the wording
      “Generate one with the metaprompt recipe from the Claude Cookbook.”
      ↩︎ Diagnose before you edit the wording
      “you can sometimes improve latency and cost more easily by choosing a different model”
      ↩︎ Exam trap 1
    4. 5.
      “Test each addition to see if it actually improves results.”
      ↩︎ Diagnose before you edit the wording
      “Don't over-engineer: Longer, more complex prompts are NOT always better.”
      ↩︎ Diagnose before you edit the wording
      “If the data is insufficient to draw conclusions, say so rather than speculating.”
      ↩︎ Constraining scope, format and behavior without over-constraining
      “Often, being explicit about what perspective you want is more effective”
      ↩︎ Constraining scope, format and behavior without over-constraining
      “If any metric is not clearly stated in the report, use null rather than guessing.”
      ↩︎ Conflicting and over-prescriptive rules
      “Pro tip: Start simple and add complexity only when needed. Test each addition to see if it actually improves results.”
      ↩︎ Checkpoint
    5. 7.
      “Precisely define the output format you want using JSON, XML, or custom templates”
      ↩︎ Conflicting and over-prescriptive rules
      “Precisely define the output format you want using JSON, XML, or custom templates so that Claude follows every output formatting element you require.”
      ↩︎ Checkpoint
    6. 8.
      “led the model to hold back about half of its routine requests to colleagues and other teams.”
      ↩︎ Conflicting and over-prescriptive rules
      “A rule that limits scope doesn’t cancel it.”
      ↩︎ Conflicting and over-prescriptive rules
      “You are the owner and long-term maintainer of this codebase, and its design is up to you.”
      ↩︎ Conflicting and over-prescriptive rules
      “If you also write approval rules into your system prompt, be specific about the actions you want double-checked.”
      ↩︎ Checkpoint

    Ready to test yourself?

    Practise Anthropic Claude Certified Architect – Professional (CCAR-P) - December 7 onwards in quiz mode.

    Spotted a mistake, or was something unclear? Tell us.