Run One Prompt Over 200 Rows, With an Audit Trail You Can Check

Two hundred rows is not two hundred conversations.

Somewhere in most teams there is a spreadsheet nobody wants to open. Two hundred incident notes to categorise. A year of intake records with a date buried in free text. The prompt that handles one of them already exists and works. What is missing is a way to run it two hundred times and then say, with evidence, whether the result is safe to use.

Before reading this. You need a prompt that already works on a single record and a small test set you trust. If you do not have one, build it first with Build a Golden Test Set Before You Trust an AI Workflow.

After reading this article, you will be able to. Lock an output shape a machine can check, run one prompt across a few hundred rows with a per-row record you can trace back, size a verification sample honestly, and tell a systematic failure apart from a scattered one.

Why does a batch fail differently from a single answer?

In a chat window you read every answer. That is the entire quality control system, and it works, because you are the reviewer and the sample size is one. At two hundred rows the reviewer disappears. The failure mode changes from a wrong answer to a wrong answer nobody looked at, and it usually surfaces inside a report.

Two shortcuts get taken at this point. Both are worse than they look.

The first is pasting all two hundred rows into one window and asking for a table back. Anthropic's own documentation explains why that degrades: "As token count grows, accuracy and recall degrade, a phenomenon known as context rot. This makes curating what's in context just as important as how much space is available." A single answer covering two hundred rows also carries one identifier, so a wrong cell cannot be traced to the reasoning that produced it.

The second is running two hundred chats by hand. That preserves the clean window and destroys the record. Nobody logs a prompt version at three in the afternoon on row 147.

Batching solves both, because each row becomes its own request with its own identifier and its own uncluttered context. It is also cheaper. Anthropic's Message Batches API and OpenAI's Batch API both run asynchronously at 50 percent of standard pricing, and two hundred rows sits far inside their limits. Anthropic caps a batch at 100,000 requests or 256 MB, OpenAI at 50,000 requests and a 200 MB input file, both as at 26 September 2026. On Claude Sonnet 5 at $2 per million input tokens and $10 per million output tokens, a two hundred row job at roughly 800 input and 250 output tokens per row costs about 82 US cents run normally, and about 41 cents batched, as at 26 September 2026.

Lock the output shape before you run anything

A batch is only checkable if every result has the same structure. That is not a stylistic preference. It is what lets a script look at all two hundred rows while a human reads a few.

On an API you can make the structure a guarantee rather than a request. Anthropic's structured outputs feature constrains the response to a declared schema through constrained decoding, and the documentation states the point plainly: "Structured outputs constrain Claude's responses to follow a specific schema, ensuring valid, parseable output for downstream processing." The promises are narrow. Always valid, so no parsing errors. Type safe, so the declared fields exist with the declared types.

Read what is not promised. The same documentation lists what a schema cannot express: numerical constraints such as a minimum or a maximum, string constraints such as a length, and recursive structures. So the schema will happily accept a compensation amount of forty million dollars and a decision date in 1804. Structured outputs guarantee shape. They never guarantee correctness. A team that conflates the two has bought a false sense of assurance, which is the single most expensive mistake available in this workflow. If output contracts are new to you, Structured Outputs: Get the Same Shape Every Time covers the design side properly.

Without an API you can get most of the way there with a fixed template, an explicit list of allowed values, and a required token for absent data. The prompt below is the shape, not the wording, and it should be adapted to your task.

You are extracting fields from one record. Return only the fields named below.

Record identifier: [ROW_ID]
Record text:
[ROW_TEXT]

Return exactly these fields, in this order, one per line:
record_id: copy [ROW_ID] exactly
category: one of INTAKE, ASSESSMENT, DECISION, CLOSED
decision_date: YYYY-MM-DD, or NOT_FOUND
amount_aud: a number with two decimals, or NOT_FOUND
review_flag: YES or NO
review_reason: one sentence, or NONE

Do not infer a category that is not written in the record.
Do not calculate, convert or estimate an amount.
Do not resolve a conflict between two dates. Set review_flag to YES and name
both dates in review_reason.
Do not add commentary, headings, or any field not listed above.

The NOT_FOUND token matters more than it looks. Without it, a missing date produces either a blank the parser silently drops or an invented one nobody notices. With it, absence is a value you can count.

Pilot the contract on ten real rows before you run the full set. The batch is not where you discover the prompt was wrong.

What has to be in the row log?

An audit trail is not the spreadsheet of answers. It is the ability to take any single output back to the exact input, prompt and model that produced it, six months later, in front of someone who did not run it.

Six fields are the minimum for every row:

• The row identifier, carried from your source data and never regenerated.
• The exact input text sent, or a stable reference to it, so the run can be reproduced.
• The prompt version, as a number you increment on any change, however small.
• The model identifier including its version, because behaviour moves between versions.
• The run identifier and timestamp, so two runs over the same data cannot be confused.
• The raw output as returned, alongside the parsed fields, because a parsing bug is invisible once the raw text is gone.

Add two more when a person touches a row: who reviewed it and what they decided. A row that was corrected by hand is a different artefact from one that passed, and the difference has to survive into the record.

Batching also carries a governance consequence that the same work done interactively does not. Asynchronous processing requires server-side storage, so requests and results sit with the vendor by design. Anthropic states that batch results are available for 29 days after batch creation and that request and response data is stored for up to the same period, with a deletion endpoint available. OpenAI states the batch output file is deleted 30 days after the batch completes. Both figures are as at 26 September 2026. If the rows contain anything sensitive, design around that footprint before the run, not after, and Privacy-Safe AI for Regulated Work sets out the de-identification step that belongs upstream of it.

How many rows do you actually need to read?

Two checks do different jobs, and confusing them is how teams overclaim.

The shape check is free and covers everything. A script reads all two hundred rows and asks whether every required field is present, whether every category is one of the four allowed values, whether every date parses, and how many fields came back NOT_FOUND. That is 100 percent coverage of a real question.

The substance check is expensive and covers a sample. Somebody has to open the source record and decide whether the extracted answer is true. This is where the honest arithmetic lives, and it is less generous than most people expect. Hanley and Lippman-Hand set out the shortcut in JAMA in 1983: "This 'rule of three' states that if none of n patients shows the event about which we are concerned, we can be 95% confident that the chance of this event is at most three in n."

Applied to a batch, it means a clean random sample of 30 does not tell you the run is clean. It tells you that a true error rate of about 10 percent has not been ruled out. Their own table gives the ladder: zero errors in 50 bounds the rate at about 6 percent, and zero in 100 bounds it at about 3 percent. If the output feeds a decision where one wrong row in ten is unacceptable, 30 is not a sample, it is a gesture.

Then stop sampling only at random. Target half the sample at rows the run itself flagged as unusual: everything with a NOT_FOUND, everything with review_flag: YES, the longest and shortest inputs, and every row in the least common category. Only the random half licenses the bound. The targeted half finds the mechanism.

Systematic or scattered, and why the difference decides your next move

When the sample turns up errors, the first question is not how to fix them. It is whether they share a cause.

A systematic failure clusters. Every row in one category is wrong. Every row over a certain length loses its last field. Every row processed after a model version changed behaves differently from the ones before it. The cause sits upstream in the prompt, the schema, a column mapping or the model itself, and the fix is made once and the whole batch re-run.

A scattered failure does not cluster. Six unrelated rows are wrong for six unrelated reasons: an empty field, an unusual date format, a note written in shorthand, a genuinely ambiguous case. The cause is input variance, the fix is by hand, and each case is added to the golden test set so the next version of the prompt has to survive it.

Telling them apart takes three sorts of the failing rows: by category, by input length, and by position in the run. If the failures pile up on any one of those axes, treat it as systematic. If they scatter across all three, treat it as variance. Getting this backwards is costly in both directions. Hand-patching 40 rows that one line of the prompt would have fixed wastes an afternoon and ships a prompt that will do it again next month. Rewriting a prompt because two odd rows failed makes it worse for the 198 that were fine.

Finish by writing down what the record actually supports. Not "we checked it". Something closer to this: shape validated on all 200 rows, substance verified on 42 rows, half random and half targeted at flagged and outlier rows, zero errors in either, so the random half bounds the rate at roughly 14 percent with 95 percent confidence, on prompt version 1.4 and model version as logged. That sentence is defensible in a review. The other one is not.

Try this

Take one repeated extraction task and write its output contract before you touch the data. Name every field, the allowed values for each, and the token the model must return when a field is absent. Run it over ten real rows and check one thing only: whether the shape held on all ten. Ten rows will tell you more about the contract than two hundred will tell you after the fact.

Where to go next

• Build a Golden Test Set Before You Trust an AI Workflow
• Structured Outputs: Get the Same Shape Every Time
• Privacy-Safe AI for Regulated Work
• Your AI Passed the Pilot. Production Is a Different Test.

Bottom line

A prompt that works once has been tested by a reader; a prompt that runs two hundred times has been tested by nobody unless you build the checks in. Lock the output shape so a machine can validate every row, keep a per-row log that carries the input, the prompt version, the model and the raw output, and split verification into a shape check that covers everything and a substance sample that covers what you can honestly claim. Then read the failure pattern before touching the prompt, because a systematic failure and a scattered one call for opposite responses. Write down the bound, not the reassurance.

Do this Monday

• Pick one repeated task and write its output contract as named fields, allowed values and a required token for absent data.
• Run the contract over ten real rows and fix it before scaling, because the batch is not where you discover the prompt is wrong.
• Set up the row log with six fields: row identifier, exact input, prompt version, model identifier, run timestamp and raw output.
• Decide the error rate the decision can tolerate, then size the verification sample from the rule of three rather than from habit.
• Sort every failing row by category, by input length and by run position before you change a single word of the prompt.

References

1. Anthropic. Structured outputs. Platform documentation. https://platform.claude.com/docs/en/build-with-claude/structured-outputs
2. Anthropic. Batch processing. Platform documentation. https://platform.claude.com/docs/en/build-with-claude/batch-processing
3. Anthropic. Context windows. Platform documentation. https://platform.claude.com/docs/en/build-with-claude/context-windows
4. Anthropic. Pricing. Platform documentation. https://platform.claude.com/docs/en/about-claude/pricing
5. OpenAI. Batch API. Developer platform guide. https://developers.openai.com/api/docs/guides/batch
6. Hanley JA and Lippman-Hand A. If nothing goes wrong, is everything all right? Interpreting zero numerators. JAMA, 1 April 1983, volume 249, number 13, pages 1743 to 1745. https://jamanetwork.com/journals/jama/fullarticle/385438