Get a Clause-Level Diff, Not a Summary of Changes

A model asked to compare two versions of a contract will tell you what changed. It will not reliably tell you what vanished.

Before reading this. Structured Outputs: Get the Same Shape Every Time is the real prerequisite, because everything here depends on an output contract with a fixed status field. Prompt Engineering Fundamentals helps, but is not essential.

After reading this article, you will be able to. Segment two document versions into an aligned, numbered comparison set, write a per-segment prompt that forces an explicit unchanged value rather than silence, and run the three arithmetic checks that catch a dropped or invented segment before anyone reads a finding.

Why does a model summarise when you asked it to diff?

Paste two versions of an agreement into a chat window, ask what changed, and you will get a summary. It will be readable, organised by theme, and incomplete. That is not a defect in the model. It is a faithful answer to the question you asked.

Two mechanisms are doing the damage.

The first is task ambiguity. Compare is a request for a characterisation. Diff is a request for an exhaustive mapping, in which every unit of one version is matched to a unit of the other and the outcome of every match is stated. Ordinary language does not distinguish the two, so the model resolves the ambiguity toward the more conversational task and hands you prose.

The second is length. Anthropic documents it in its own platform guidance, which is a vendor admitting against interest. As token count grows, accuracy and recall degrade, a phenomenon Anthropic names context rot, and curating what sits in context matters as much as how much space is available. Liu and colleagues found a related effect in Lost in the Middle. Performance is often highest when relevant information sits at the beginning or the end of the input context, and degrades significantly when a model must reach into the middle of a long one.

Then there is the failure neither mechanism explains, and it is the one that matters most.

A change is a presence. A deletion is an absence. When a clause is amended, the old text and the new text both sit in the window, and a model that notices either has something to work from. When a clause is removed, the second version contains nothing at all that points at it. The only route to finding it is to have held the clause from the first version and gone looking for its counterpart. Nothing in the second document will prompt that search.

Published long-context scores do not measure this. Multi-needle retrieval tests plant facts inside a long document and check whether the model finds them again. Finding a planted fact and noticing a missing one are opposite operations, and only the first has a number attached.

The window may also be smaller than the marketing suggests. As at 26 September 2026 Claude's plan comparison lists the context window on every plan as up to 1 million tokens, varying by model, so the headline number is a ceiling rather than a promise. Gemini without a paid plan runs on 32,000 tokens. A single long agreement can pass that 32,000 ceiling before your instructions are counted.

Run the deterministic tool first

No model should touch the documents until a real diff has run.

If both versions are Word files, Word's legal blackline option compares them directly. Microsoft's own description is plain: the option compares two documents and displays what changed between them in a third new document, and by default the documents being compared are not changed. You can set it to show changes at character level or word level. It is deterministic, it is already licensed, and it cannot invent a clause that was never there.

Read its output before writing a single prompt. On an ordinary comparison much of the work is finished at that point, at no cost.

Then notice where it stops. A comparison that aligns on characters or words breaks precisely where text has moved rather than changed:

• A schedule renumbered, so every clause number after it shifts and the whole tail marks as changed
• A document round-tripped through PDF, so line breaks, hyphenation and quotation marks now differ everywhere
• A clause relocated from section 8 to section 12, which reads as one deletion and one unrelated insertion
• A defined term renamed, which reads as hundreds of tiny edits carrying a single decision
• Two clauses merged into one, which no character comparison can describe

In each case the output is technically accurate and practically useless. That is the gap a model fills. Its job is not to find what the tool already found. Its job is the semantic layer: aligning segments the tool could not, and saying what a change does to an obligation.

Draw that boundary before you start. A workflow that hands the mechanical comparison to a model is paying for uncertainty it could have avoided.

Segment before you compare

The method is unglamorous, and it works because it is unglamorous.

Choose the unit. Clause, sub-clause, numbered paragraph or heading-level section. One rule, applied identically to both versions. The unit has to be small enough that a whole one fits comfortably inside a single comparison, and large enough to carry one complete obligation. For most commercial agreements the numbered sub-clause is right. For a policy, the heading-level section usually is.

Number both versions independently. Version A becomes A-1 through A-n. Version B becomes B-1 through B-m. Resist the urge to renumber one to match the other, because any mismatch between n and m is itself evidence.

Build the alignment table before any comparison runs. Every A segment gets exactly one B counterpart or the value NONE. Every B segment gets exactly one A counterpart or NONE. Match on heading text and clause identity, never on position. This is the step that finds deletions and insertions, and it is bookkeeping rather than judgement, which is exactly why it belongs to you and not to the model.

Compare pair by pair. One prompt, one pair, identical wording every time.

Then read. Not before.

Segmenting also disposes of the length problem outright. Each comparison holds two clauses rather than two contracts, so the window never fills and context rot never engages. OpenAI arrives at the same place from the other direction: its structured outputs guide, advising what to do when outputs contain mistakes, recommends splitting tasks into simpler subtasks.

One order-of-operations point before you upload anything. If either version carries personal information, de-identify it first, not afterwards, and use consistent bracketed placeholders so the pairing survives the substitution. Privacy-Safe AI for Regulated Work covers that discipline in full.

What does an explicit unchanged value actually buy you?

It buys the difference between silence and a statement.

Ask for "the differences" and an unchanged clause produces no output. Neither does a clause the model skipped. Neither does a clause it misread as identical. Neither does a clause that never reached it at all. Four distinct states collapse into one indistinguishable absence, and an absence cannot be audited.

Force a status on every segment instead, drawn from a closed set: unchanged, amended, added, deleted or unresolved. Every segment now produces a row. Silence becomes a countable event rather than an assumed pass, and the unresolved value gives the model somewhere honest to put what it cannot handle, which is the only way you will see it.

You are comparing one numbered segment from two versions of the same
document. Compare only the segment pair given below. Do not consider any
other segment, and do not summarise either document.

SEGMENT ID: [SEGMENT_ID]
VERSION A TEXT: [VERSION_A_TEXT]
VERSION B TEXT: [VERSION_B_TEXT]

Return one JSON object with exactly these fields:
  segment_id     the value of SEGMENT_ID, copied verbatim
  status         one of unchanged, amended, added, deleted, unresolved
  changed_spans  a list of objects, each holding the exact words removed
                 from A and the exact words added in B, quoted verbatim
  effect         one sentence on what the change does to the obligation,
                 or the single word NONE when status is unchanged
  confidence     high, medium or low

Use unchanged only when A and B match word for word.
Use added only when VERSION A TEXT is empty.
Use deleted only when VERSION B TEXT is empty.
Use unresolved when you cannot complete the comparison for any reason.

Do not paraphrase any text you quote.
Do not infer intent, motive or negotiating position.
Do not comment on any segment other than [SEGMENT_ID].
Do not state a legal effect, a compliance conclusion or a recommendation.
Do not return prose outside the JSON object.

Where the platform supports it, enforce that shape rather than requesting it. Anthropic's structured outputs constrain responses to follow a specific schema, ensuring valid, parseable output for downstream processing. OpenAI's equivalent ensures the model will always generate responses that adhere to your supplied JSON Schema, so you do not need to worry about the model omitting a required key or hallucinating an invalid enum value.

Then hold the limit in view. It is the most misread point in this area. Schema conformance guarantees shape. It does not guarantee truth. OpenAI says as much in the same guide: structured outputs can still contain mistakes. A schema can require that a status field exists and holds one of five permitted words. Nothing in it can make the chosen word correct. The status field is scaffolding for checking, not a replacement for checking.

The three checks that catch a dropped segment

Run all three before anyone reads a single finding. Each is arithmetic, and none of them depends on judgement.

The count check. Rows out must equal pairs in. Send 214 pairs, expect 214 rows. Two hundred and thirteen is a silent drop.

The identity check. Every segment identifier returned must appear in your input list, and every input identifier must appear exactly once in the output. Match one to one, not set against set, because a duplicated row and an omitted row net to zero on a count and stay invisible to it.

The boundary check. Count the rows returned as deleted and as added. Those totals must equal the NONE entries you recorded in the alignment table. If the table showed three A segments with no counterpart and the output reports one deletion, the disagreement is the finding, and you go back to the table rather than forward to a report.

Only then start reading, in this order: every unresolved row, then every low-confidence row, then a fixed random sample of the rows marked unchanged. Sample the unchanged rows deliberately. Unchanged is the status an inattentive pass over-produces, which makes it least likely to be checked and most likely to be quietly wrong.

If you hold a pair of versions where you already know every change, run the whole method over it as a control before trusting it on live work. That is a golden test set applied to a one-off comparison, and it is the cheapest assurance in the workflow.

What none of the checks can do is tell you what a change means. Whether a shortened cure period matters, whether a relocated indemnity still bites, whether a redefined term has quietly widened a liability: those are judgements. They stay with the practitioner who signs the advice, and no status field will ever carry them.

Try this

Take two versions of one short policy, no more than fifteen sections. Number both independently, build the alignment table by hand, and deliberately delete one whole section from the second version before you start. Run the per-segment prompt over every pair, then run the count, identity and boundary checks. Confirm that the deleted section came back as a deleted row rather than as silence. If it did not, the segmentation rule is wrong before the prompt is.

Where to go next

• Structured Outputs: Get the Same Shape Every Time
• Build a Golden Test Set Before You Trust an AI Workflow
• Privacy-Safe AI for Regulated Work
• Prompt Engineering Fundamentals: The 2026 Update

Bottom line

A summary tells you what someone thought was interesting about the changes; a diff tells you what happened to every clause, including the ones nothing happened to. The gap between those two is where a deleted obligation goes missing, because deletion is the one change that leaves no trace in the document you are reading. Run the deterministic comparison first, segment and align both versions before a model sees either, force a status on every single segment so that silence becomes a countable event, and check the arithmetic before you check the substance. Then do the part no schema can do, which is deciding what the changes mean.

Do this Monday

• Run the deterministic comparison on your next version pair before writing any prompt, and read its output first.
• Pick one segmentation unit for that document type and write it down so the next comparison uses the same rule.
• Build the alignment table by hand, marking every segment with no counterpart as NONE.
• Add the closed status set to your comparison prompt so every segment returns a row, including unchanged ones.
• Run the count, identity and boundary checks before anyone reads a finding, and record the three totals with the output.

References

1. Anthropic. Context windows. Platform documentation. https://platform.claude.com/docs/en/build-with-claude/context-windows
2. Anthropic. Structured outputs. Platform documentation. https://platform.claude.com/docs/en/build-with-claude/structured-outputs
3. Anthropic. Pricing. Claude plan comparison. https://claude.com/pricing
4. OpenAI. Structured Outputs. API documentation. https://developers.openai.com/api/docs/guides/structured-outputs
5. Microsoft. Compare document differences using the legal blackline option. Microsoft Support. https://support.microsoft.com/en-us/office/compare-document-differences-using-the-legal-blackline-option-dbfc7351-4022-43a2-a0c4-54d1898702a0
6. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F. and Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics. https://arxiv.org/abs/2307.03172
7. Google. Gemini Apps limits and upgrades for Google AI subscribers. Google Help. https://support.google.com/gemini/answer/16275805