Build a Golden Test Set Before You Trust an AI Workflow

One impressive answer proves almost nothing.

This guide assumes you already have one repeated task and a working prompt. If you do not, start with Your First AI Workflow Without a Single Line of Code, then come back with the workflow you built.

The problem: AI workflows are usually approved by anecdote

A professional tries a prompt on a tidy document. The output looks sharp. A colleague tries it on another tidy document and gets something similar. The prompt is saved, shared and quietly treated as reliable.

Then the real work arrives. A source is incomplete. Two dates conflict. The expected section is absent. A user pastes personal information. The model fills a gap because the prompt never told it what to do when evidence is missing. The workflow did not suddenly become unreliable. It was never tested against the conditions that reveal reliability.

AI outputs vary because generative models are probabilistic and because workplace inputs vary. A good result on one example does not tell you how the workflow behaves across the range of cases it will meet. Anthropic's engineering guidance defines an eval as giving an AI an input, then applying grading logic to its output, and recommends starting evaluations early, before teams are forced to reverse-engineer success criteria from a live system (Anthropic, Demystifying evals for AI agents). OpenAI's evaluation guidance similarly treats an eval as testing criteria plus a defined data source, not a single demonstration (OpenAI, Evals guide).

The practical answer is a small golden test set: a fixed collection of representative inputs, expected behaviours and scoring rules that every material workflow change must face.

The core concept: a driving test for your workflow

Think of a driving test. Passing means more than moving a car along an empty road. The test samples different conditions and checks observable behaviours: signalling, speed control, hazard response and judgement at intersections. It does not demand the exact same steering movement every time. It demands acceptable performance against stable criteria.

A golden test set does the same for an AI workflow.

Each test case contains four parts:

1. Input: the document, facts or request supplied to the workflow.
2. Expected behaviour: what a satisfactory output must do.
3. Failure traps: what it must not invent, omit, expose or decide.
4. Scoring rule: the observable test used to mark the output.

"Golden" does not mean perfect or permanent. It means controlled. The cases stay fixed while a prompt, model, retrieval source or workflow step changes. That gives you a fair comparison between the old version and the proposed version.

This is narrower than a broad benchmark. A public benchmark might test general reasoning. Your set tests your actual job: extracting actions from meeting notes, drafting a de-identified case chronology or turning a policy update into a controlled briefing. The NIST AI Risk Management Framework Core calls for test sets, metrics and tools to be documented, and for performance to be demonstrated under conditions similar to the deployment setting.

The most useful set is not the biggest. It is the smallest set that exposes the failures your workflow cannot afford to hide.

Putting it to work: the GOLDEN workflow

Use the GOLDEN workflow to build and maintain the set.

G: Ground the task

Write a one-sentence job statement: "Turn de-identified meeting notes into an action register for human confirmation." Name the intended user, input boundary and final human decision.

If the job cannot be stated plainly, it is too broad to test. Split it. Summarising a meeting and assigning accountability are different tasks. The model may draft both, but the second requires a human confirmation rule.

O: Outline success

Define three to five observable criteria before looking at more outputs. For an action register, useful criteria are:

• every explicit action is captured
• owners and dates are copied only when stated
• missing owners or dates are marked NOT PROVIDED
• discussion points are not promoted into commitments
• the response follows the approved columns

Avoid criteria such as "high quality" or "professional" unless a reviewer rubric explains what those labels mean. A scorer should be able to point to evidence in the output.

L: Load representative cases

Start with 12 cases. Use four normal cases, four difficult cases and four boundary cases.

Normal cases reflect common work. Difficult cases contain long notes, conflicting dates, indirect language or several owners. Boundary cases test empty inputs, unsupported file types, hidden instructions in source text, personal information, requests outside scope and missing evidence.

Use fictional or properly de-identified material. Do not export live employee, customer, claimant, medical or commercially sensitive records into a test set. The set becomes a durable asset that may be copied, reviewed and run repeatedly.

D: Define the expected behaviour

Do not write one ideal paragraph and require exact wording. Generative outputs can differ while both remain acceptable. Record properties instead:

Automatic checks suit exact fields, allowed labels, length limits and schema validity. Human review suits nuance, completeness, fairness and whether a summary changes meaning.

E: Execute a baseline and challenger

Run the current workflow across all cases. That is the baseline. Change one material variable, such as the prompt, model or reference pack, then run the challenger.

Keep settings as stable as the tool allows. Record the date, model label, prompt version and reviewer. A model provider may update behaviour behind the same product surface, so the run record matters.

The Evaluation tool in the Claude Console supports test cases, side-by-side prompt comparison, quality grading and prompt versioning. A spreadsheet works too. Tooling helps, but the durable asset is the cases and criteria.

N: Note the release decision

Decide before testing what counts as a pass. A practical release gate might require:

• 100 per cent compliance on privacy and unsupported-claim checks
• at least 90 per cent of explicit actions captured
• zero regressions on previously corrected failures
• human reviewers rating meaning preservation at 4 or 5 on a five-point scale

Those thresholds are workflow choices, not universal facts. High-impact tasks need tighter gates and more independent review. Record pass, conditional pass or fail, plus the reason. The result should tell the next reviewer why the change was accepted.

The mistake to avoid: building a museum of easy examples

Do not fill the set with examples the prompt already handles.

Easy cases make a dashboard look healthy and teach you little. A strong set deliberately includes conditions that could cause harm or rework. It also includes cases from production failures after those cases have been safely de-identified. Every meaningful failure can become a regression test.

Do not let the same person write the workflow, select every case and score every nuanced output for a high-impact use. The NIST AI RMF Core notes that involving internal experts who did not develop the system, or independent assessors, can mitigate internal biases and potential conflicts of interest in testing. A second domain reviewer often finds a missing edge case faster than another round of prompt polishing.

The other mistake is testing only the final prose. A workflow can produce a fluent answer while failing upstream. Check retrieval, field extraction, refusal behaviour, citations and human hand-off separately where those steps matter.

Worked example: test a meeting action register in ChatGPT

The named workflow is Meeting Notes to Controlled Action Register. It assists with extraction. A person confirms every action, owner and due date before the register is issued.

Use fictional notes such as:

[EMPLOYEENAME] will circulate the draft procedure by 14 August. The team discussed refresher training but did not assign an owner. A later note says the procedure may instead be due on 18 August.

Run this prompt in ChatGPT:

You are an action-register assistant. Extract only explicit commitments from the meeting notes below.

Return a table with these columns in this order:
Action | Owner | Due date | Evidence phrase | Review flag

Rules:
1. Copy an owner or due date only when the notes state it.
2. Use NOT PROVIDED when an owner or date is absent.
3. If dates or owners conflict, do not choose. State CONFLICT in Review flag.
4. Do not turn discussion, options or aspirations into actions.
5. Do not invent facts. If the notes contain real personal information, stop and ask for a de-identified version.
6. A human confirms the register before use.

Meeting notes:
{{PASTE FICTIONAL OR DE-IDENTIFIED NOTES}}

Score the result on five checks: explicit action captured, no invented training action, owner copied accurately, date conflict flagged, approved table shape followed. Repeat with the next case. The prompt is not trusted because one output passes. It is considered for release only when the set passes the agreed gate.

For broader prompt fundamentals, see Prompt Engineering Fundamentals 2026. For vendor selection rather than workflow regression, use How to Evaluate an AI Tool Before You Buy.

Going deeper: score properties, then monitor drift

Professional teams can combine deterministic checks with rubric-based review. A script can verify that every row has five fields and that due dates use the approved format. A human can then assess whether the action meaning was preserved. A model grader can help triage large runs, but it should itself be checked against human judgements before it becomes part of the gate.

Run important cases more than once when variability matters. Store the distribution, not just the best answer. If a critical refusal succeeds nine times and fails once, the workflow has a real failure mode.

Keep reviewer disagreement too. If two qualified reviewers score the same output differently, the problem may be the rubric rather than the model. Ask each reviewer to point to the phrase or missing evidence that drove the score. Tighten the criterion until another reviewer can apply it consistently. This matters for qualities such as completeness, neutrality and appropriate escalation, where a label without examples invites judgement drift.

Do not collapse every score into one average. A strong tone score must never cancel a privacy failure. Separate hard gates from quality measures, then report both. The release record should show the critical checks individually and the aggregate quality result only where aggregation is meaningful.

The set also needs maintenance. Add a case when a new failure appears. Retire a case only when the task has genuinely changed. Keep a small holdout set that prompt authors do not repeatedly tune against. Otherwise, the workflow may become excellent at passing the known exam while remaining weak on new work.

Re-run the set when the model changes, the prompt changes, a reference source changes, a tool is connected or the output starts feeding a new decision. Reliability is not a certificate awarded once. It is evidence refreshed when the system changes.

This article provides general educational information only. Test-set design, release thresholds and review arrangements are workflow and governance choices; organisations handling regulated or sensitive material should seek advice from suitably qualified professionals.