A tidy demonstration can miss the failure that matters
A team demonstrates a new workflow with three complete records. Every identifier is unique, every date is in order and every attachment reference resolves. The output looks sensible. When the workflow meets a missing field the next week, nobody knows whether it should stop, ask a question or continue with a warning.
The demonstration did not exercise that condition. Adding more equally tidy records would not necessarily help. The useful next step is to specify the difficult situations the workflow must handle, then construct inputs that expose them. Test data should follow the question being investigated, rather than merely resemble a normal-looking spreadsheet.
AI can draft these fixtures from a clear specification. A fixture is a controlled input with a known purpose and an expected result. It can be a small table, document or structured record. The person designing the test remains responsible for the schema, the intended rule and the answer key.
This article uses invented service-request records. It does not derive synthetic data from a real dataset or claim statistical representativeness. That narrower approach suits a practical demonstration of validation and routing. A model intended to predict population outcomes would require a different evaluation design and evidence beyond these five cases.
Begin with the workflow contract
Write down what the workflow receives and what it is permitted to produce. In this example, it receives a request ID, a submitted date, a due date, a request type and an attachment reference. It may organise the request and identify input problems. It may not silently repair missing facts or decide the underlying business request.
Next, state the validation rules. The request ID is required and must be unique within the test batch. The submitted date cannot be later than the due date under this fictional workflow's rule. A non-empty attachment reference must match an entry in the supplied attachment manifest. These are constructed rules for the tutorial, not universal business requirements.
Define the response to each failure. A missing required field produces a hold for clarification. A date conflict produces a hold with both dates preserved. A duplicate ID produces a collision report rather than an automatic merge. A broken attachment reference produces a hold until the reference is resolved or the owner changes the requirement.
A precise failure response makes the test useful. If the answer key simply says "handle appropriately", almost any polished output can appear acceptable. The reviewer needs to know which facts must remain unchanged, which uncertainty must be visible and which action the system must not take.
Construct records without borrowing a person's story
Use a schema and an invented scenario as the starting point. Avoid taking a real employee record and replacing the name, or asking a model to imitate a distinctive case. Dates, roles and combinations of events can still identify people even when obvious identifiers have been removed.
The OAIC's de-identification guidance explains that removing names, addresses or other direct identifiers alone may not result in de-identification for the purposes of the Privacy Act, and that re-identification risk must be assessed in context. That source is being updated in relation to later privacy-law changes, so it is used here for the limited conceptual distinction, not as a current security-law summary. Source: OAIC
For the teaching pack, choose neutral role labels and simple invented dates. Use an explicit synthetic label at the file and record level where practical. A record does not need a plausible person's name to test whether an identifier is missing. Avoid adding biographical detail that has no bearing on the condition being exercised.
Inspect generated text for accidental personal references or distinctive real-world facts. The instruction to invent data is a control in the process, not proof that every output is safe. If a model introduces an unnecessary narrative, remove it and regenerate from the minimal schema rather than trying to anonymise a story that was never needed.
Build five cases with one purpose each
The worked example is a fictional private-sector internal service desk. It uses five records labelled T01 to T05. These labels identify test cases and are separate from the request IDs inside the records. That distinction lets a test case retain its own identity even when the request ID is deliberately missing or duplicated.
T01 is the normal case. Its request ID is R-001, its submitted date is 1 September and its due date is 5 September 2026. It concerns a meeting-room setup and references document D-01, which exists in the manifest. The expected result is ready for human routing, with the source fields preserved.
T02 omits the required request ID. Its other fields are ordinary. The expected result is a hold identifying the missing ID. The workflow must not create R-002 merely because that would make the sequence convenient. Generating an identifier is a separate business action that this test has not authorised.
T03 has a submitted date of 8 September and a due date of 6 September. Under the fictional rule, that is a conflict. The expected result preserves both values and asks for confirmation. Reversing the dates would hide the defect and falsely imply knowledge of what the requester intended.
T04 repeats R-001 while describing a different request. It deliberately collides with T01 in the same batch. The expected result identifies both test-case locations and holds them for resolution. It does not discard T04 or assume that matching IDs prove identical requests. The collision also changes the batch disposition of T01.
T05 refers to D-99, which is absent from the attachment manifest. Its other required fields are valid. The expected result is an unresolved-reference hold. A model must not substitute D-01 because that is the only available document or invent a description of the missing attachment.
The answer key therefore distinguishes isolated-case and batch behaviour. T01 is valid when tested alone, but participates in the duplicate-ID issue when T04 is included. This is an intentional interaction. Recording it avoids the common mistake of calling T01 both passed and held without explaining the evaluation context.
Give the generator constraints and forbidden shortcuts
Ask the model to produce exactly the five cases from the scenario specification. Supply the field names, permitted formats, manifest and expected defect for each case. Require the output to retain separate test-case IDs and request IDs. Do not ask for realistic data in general terms and hope the desired defects appear.
Specify which fields must remain valid in each defective case. If the missing-ID case also contains an invalid date and an unknown request type, a failed result becomes harder to interpret. A focused fixture lets the reviewer connect the observed behaviour to the condition being tested. More complex combinations can be added later deliberately.
List the shortcuts that would invalidate the exercise. The generator must not fill a deliberate blank, deduplicate records, repair date order or create the missing attachment. These are precisely the behaviours the test is designed to expose. A helpful-looking correction can remove the condition before the workflow ever sees it.
Keep the expected results outside the input supplied to the workflow under test where practical. A tool that can read its answer key may reproduce the desired response without demonstrating that it handles the input. The human reviewer retains the key and compares it with the observed output after the run.
Inspect the fixtures before testing the workflow
First check the fixture itself against the specification. There must be five test cases with the intended defects, not five cases that merely contain the requested labels. Verify dates, identifiers and manifest membership directly. A model-generated answer key is not independent evidence that the generated inputs are correct.
Then check for unintended defects. T03's request type should remain valid so the date conflict is the intended reason for the hold. T05 should use a genuinely absent document ID. T04 must duplicate the exact request ID used by T01, including its representation, if exact-identifier collision is the rule being tested.
Keep a short fixture validation record. It can state which values were inspected, who reviewed them by role and which version was accepted for testing. If code checks the structure, retain the check and its result. Do not describe an automated validation as a human review or a text inspection as a production run.
In this article, the table and expected outputs are constructed teaching material. They are not observations from a separate AI service. A reader who runs the exercise must add an observed-output column and record the actual tool, version, settings and date. Blank observed results are more honest than invented successful tests.
Compare behaviour, including what the system refused to invent
Evaluate the observed output against the answer key, not just its fluency. Did T02 remain without a request ID? Were both T03 dates preserved? Did the duplicate report identify T01 and T04? Did T05 remain unresolved? These checks reveal whether the workflow respected the information boundary.
Also check whether the output communicates the problem to a person who must act. A response that says "invalid data" may technically recognise an error but provide no useful next step. The expected result should identify the field or relationship, the source value and the specific clarification needed.
Distinguish an input rejection from a system failure. A controlled hold on T03 is the intended behaviour, not a failed workflow. Conversely, a complete-looking route assignment that silently repairs T03 is a failure against this contract. The test outcome depends on the agreed behaviour, not on whether the tool produced a finished document.
Record unexpected but relevant behaviour separately. A model may identify a useful ambiguity that the original specification missed. Do not immediately change the answer key to make the run pass. Review the observation, decide whether the contract needs revision and version the change before repeating the test.
Expand coverage without claiming more than the fixtures show
Five cases cover five deliberately selected conditions. They do not establish reliability across all inputs. Consider what the real workflow adds: long text, different encodings, multiple attachments, unusual dates, access failures or conflicting instructions inside supplied documents. Add scenarios only when they address a meaningful requirement.
Use pairs where a small change should alter the result. For example, replace D-99 with a valid manifest reference while keeping T05 otherwise identical. If the unresolved-reference message persists, investigate whether the system is responding to stale context or a different rule. A controlled variation can be more informative than a large random dataset.
Keep the original cases stable as a reference set. If a workflow changes, rerun the same inputs before replacing them. Otherwise, an apparent improvement may reflect easier test data rather than better behaviour. New cases can supplement the original set with a documented reason and expected outcome.
Do not report a percentage pass rate without describing the cases and their significance. Four successful cases and one failure can mean very different things depending on the failure. A workflow that invents missing facts may be unsuitable for the intended use even if it handles ordinary formatting well.
Make the test pack reusable by another person
Package the schema, fixture table, manifest and answer key with version identifiers. Explain whether cases run individually or together. Include a blank results record and the procedure for handling failures. Another reviewer should be able to reproduce the exercise without relying on the original author's memory.
Keep business decisions outside the expected output unless the workflow has explicit authority and a verified decision specification. For this example, the furthest successful state is ready for human routing. That narrow endpoint makes it possible to test organisation and validation without pretending that a service request has been approved.
Start with a failure you need the workflow to recognise. Build the smallest invented input that exposes it, decide the correct response and inspect the fixture before running it. Synthetic test data becomes useful when its purpose and limits are clear enough for someone else to challenge.
Reusable materials
Five-case synthetic fixture pack
Constructed fictional private-sector service desk. No real people, real requests, production run or statistical realism claim.
Contract v1
Required: request_id, submitted_date, due_date, request_type. Permitted request_type values: room setup, equipment. IDs unique within batch. Submitted date must not follow due date. Non-empty attachment references must exist in manifest. Successful validation means ready for human routing, not approved.
Manifest
Fixtures
[missing] below represents an absent value, not literal text to validate. Dates are ISO calendar dates.
Answer key, kept separate from workflow input
Fixture-generation prompt
Observed-results record
No observed AI-run results are populated in this teaching pack.
Related reading
References
- Office of the Australian Information Commissioner, De-identification and the Privacy Act, published 21 March 2018; the OAIC states the page is being updated for amendments made by the Privacy and Other Legislation Amendment Act 2024: https://www.oaic.gov.au/privacy/privacy-guidance-for-organisations-and-government-agencies/handling-personal-information/de-identification-and-the-privacy-act
General information only. Not legal, compliance, financial, or professional advice.
TheAICommand. Intelligence, At Your Command.
