A Defensible Claims Audit Sample Starts With the Population, Not an AI Risk Score

An AI-selected list of unusual claims may be useful for investigation, but it cannot represent a claims management system. Start with the published population and sample bands, preserve a representative core, and keep every targeted file, substitution and audit conclusion under competent human control.

A claims audit sample is not a list of files that look interesting. It is evidence for a conclusion about a system. If the selection method overweights difficult, expensive or unusual claims, the audit may find real problems while still saying little about ordinary practice.

AI can help freeze the eligible population, apply Comcare's published sample band and expose missing coverage. It cannot certify that the population is complete, decide that a sample is representative, replace an unavailable file or rate a criterion.

For self-insured banks and insurers, and for other financial-sector organisations weighing the licence pathway, that distinction is essential. Assurance sampling tests the claims management system. It must never become a hidden claimant score or a mechanism for directing case-manager attention.

What population and sample does Comcare actually describe?

The current Safety, Rehabilitation and Compensation Act 1988 is Compilation No. 82 (the SRC Act), in force from 1 July 2026. In Part VIII, sections 108B and 108C address authorisation to manage claims, while section 108D permits the Commission to grant a licence on conditions.

Within that framework, Comcare's current Claims management system audit tool, version 5.0, dated October 2025, is used by Comcare, self-insured licensees and claims management providers to assess claims management systems. The tool says the SRCC uses it in its monitoring role. The current SRCC audit-tools page lists the tool, workbook and report template for audits after a self-insurance licence has been granted.

The audit tool starts with scope. Findings should represent the state of the relevant authority's overall claims management system, and the scope must be sufficient to produce reliable and robust findings. For an audit of the entire system and its implementation, it suggests these sample sizes:

Those bands are precise, but they are not an automated conclusion. The eligible population is claims with some form of activity within the 12 months before the audit date. The tool says the sample may be increased where multiple sites are tested. If the auditor uses discretion and adopts an alternate sampling methodology, the explanation belongs in the audit report.

The first AI task is therefore reconciliation, not scoring. Compare the frozen claims-system extract with the audit date, activity records, site and administrator data. Show inclusions, exclusions, duplicates and missing fields. A competent human auditor then approves the population and scope.

Why does a risk score weaken the audit?

A risk model answers a different question. It might rank files by cost, duration, litigation, psychological injury, medical-report volume or data irregularity. That can produce a useful targeted list, but the list is deliberately unlike the population.

The published audit tool says findings should be representative of the overall claims management system. It also says the auditor must choose an appropriate sample, and that a breadth of claims decisions should be reviewed, giving initial decisions, permanent impairment claims, death claims and funeral expenses, incapacity determinations and suspensions as examples. It does not endorse an AI risk score as a sampling method.

Use a proposed three-part architecture instead. This is an internal governance technique, not a Comcare requirement.

1. Population register. Freeze the eligible set, the extraction time, query rules and exclusions. Retain row counts and source-system reconciliation results.
2. Representative core. Apply the published band and a documented human-approved selection method across the eligible population. The Comcare tool does not prescribe random selection. If the auditor chooses seeded random, systematic or another method, record that choice and why it supports the scope.
3. Targeted overlay. Add files selected to examine a known control concern, decision type or site. Label them targeted. Do not count them as evidence that the core is representative merely because they were reviewed.

This prompt prepares the population register and candidate band, following the same human-review discipline as the scheme-specific WC prompt libraries. A human auditor must verify the extract, approve eligibility, choose the selection method and approve the final sample.

Prepare a de-identified claims audit population register for [AUDIT_ID] using only the supplied extract.

Audit date: [AUDIT_DATE]
Required fields: [CLAIM_ID], last activity date, activity source, site, administrator, decision-type indicators and source row.

Identify claims with recorded activity in the 12 months before [AUDIT_DATE]. Report included rows, excluded rows with reasons, duplicates, missing activity evidence and reconciliation differences. Apply the published Comcare population band only to show the suggested sample number.

Do not create a claimant risk score. Do not select files, decide eligibility, infer missing activity, approve the population or claim representativeness. Label every unresolved item HUMAN AUDITOR REVIEW.

Do not feed claim narratives into this stage. A selection manifest usually needs controlled metadata, not medical histories or detailed allegations. Comcare's audit tool says the privacy and confidentiality of information collected for an audit is subject to the Privacy Act 1988. OAIC guidance on commercially available AI calls for product due diligence and recommends that organisations not enter personal information, particularly sensitive information, into publicly available generative AI tools. Use de-identified fields and an organisationally approved environment.

How should coverage and substitutions be controlled?

Build the coverage matrix after the core has been selected. That ordering prevents the team from repeatedly changing the sample until every preferred category appears.

Map each selected file against relevant characteristics such as decision type, site, administrator and audit criterion. The matrix is a diagnostic. It helps the auditor see whether the sample supports the planned scope and whether a targeted overlay or documented alternate method is needed. It does not prove representativeness by itself.

Keep the overlay in a separate column and in the report narrative. A targeted file can reveal a control failure. It cannot be silently blended into the representative core to make that failure look population-wide. Equally, a clean targeted file cannot repair weak core coverage.

Substitution needs its own ledger. The Comcare tool does not prescribe a substitution method, so this is another proposed internal control. Retain the originally selected file, the reason it could not be reviewed, the person approving replacement, the selection rule used for the substitute, the replacement file and the date. Never let AI quietly swap a file because records are incomplete, access is difficult or the file may complicate the result. The same receipt-integrity discipline that protects day one at claim intake protects the selection trail here; see section 54 intake and the claim clock.

Consider this fictional, de-identified example. The frozen population for [AUDIT_ID] contains 312 claims with recorded activity in the 12 months before [AUDIT_DATE]. The published band suggests 70 claim files. A human auditor approves a documented selection method and a 70-file representative core.

The coverage matrix shows limited exposure to one decision type and a concern has already been raised about [SITE_CODE]. The auditor adds eight targeted files. The audit now reviews 78 files, but the report continues to identify 70 as the representative core and eight as the targeted overlay. When [CLAIM_ID_17] cannot be accessed, no model replaces it. The auditor records the issue and approves [CLAIM_ID_71] under the documented substitution rule.

This prompt stress-tests the manifest. A human auditor must resolve every exception and approve the final scope, substitutions and report description.

Audit this de-identified selection manifest for [AUDIT_ID].

Separate the representative core, targeted overlay and proposed substitutions. Check each row against the frozen population, approved selection rule and source row. Produce:
1. core count against the published sample band;
2. coverage gaps and concentrations;
3. targeted files incorrectly labelled as core;
4. substitutions without a retained original, reason or human approval; and
5. wording the auditor should verify for the scope section.

Do not rate an audit criterion, approve representativeness, choose a substitute, generalise a targeted finding, assess a claimant or recommend case-management action. Mark all decisions HUMAN AUDITOR REQUIRED.

Human ownership continues after selection. The October 2025 workbook has 31 criteria across five elements and requires the auditor to judge whether each criterion is met, using ratings of conformance, non-conformance, not able to verify or not applicable. It says competent personnel need SRC Act knowledge plus relevant audit training and experience, and auditors must be independent of the area audited.

The SRCC licence-compliance page confirms that Comcare monitors licence compliance on the SRCC's behalf and that reviews may occur during the developing phase, at years two or six, or in response to a concern. AI can make the selection trail easier to inspect. It cannot turn its score into regulator-endorsed evidence or replace the auditor's judgement.

Do this Monday

1. Freeze one test population. Run the 12-month activity query for [AUDIT_DATE], preserve its logic and reconcile the row count against the claims system.
2. Apply the published band. Record the suggested file count and any proposed increase for sites or scope. Require the auditor to approve any alternate methodology and report explanation.
3. Separate core from overlay. Add an immutable selection_basis field with representative core or targeted overlay. Do not permit a blank value.
4. Create a substitution ledger. Require the original file, reason, human approver and replacement rule before a substitute can enter the review set.
5. Test the coverage matrix. Use fictional claim IDs to confirm it exposes concentrations without changing the sample or issuing conclusions.
6. Lock the decision boundary. Restrict AI output to reconciliation, band lookup, coverage flags and candidate report wording. Keep scope, ratings, findings and corrective actions with competent auditors.

Bottom line

An audit sample begins with a complete, time-bounded population and an explicit purpose. AI can reconcile that population, apply Comcare's published bands and reveal gaps in the selection manifest. The auditor must approve the representative core, distinguish every targeted file, control substitutions and make every finding. An AI risk score is not a shortcut to defensible assurance.

This article is general information and education only. It is not legal advice, and it is not advice about any individual claim. Decisions under the Safety, Rehabilitation and Compensation Act 1988 are made by human decision-makers on the individual merits of each claim, and claimants have reconsideration and review rights in respect of determinations. Seek advice specific to your scheme and circumstances.

References

1. Federal Register of Legislation, Safety, Rehabilitation and Compensation Act 1988, Compilation No. 82, in force 1 July 2026. https://www.legislation.gov.au/C2004A03668/latest/text
2. Comcare, Claims management system audit tool, version 5.0, October 2025. https://www.comcare.gov.au/sites/default/files/docs/claims-management-system-audit-tool.pdf
3. Comcare, Claims management system audit workbook, version 5.0, October 2025. https://www.comcare.gov.au/sites/default/files/docs/claims-management-system-audit-workbook.pdf
4. Safety, Rehabilitation and Compensation Commission, Audit tools and templates. https://www.srcc.gov.au/current-self-insurers/audit-tools-and-templates
5. Safety, Rehabilitation and Compensation Commission, Licence compliance and performance. https://www.srcc.gov.au/current-self-insurers/licence-compliance-and-performance
6. Office of the Australian Information Commissioner, Guidance on privacy and the use of commercially available AI products. https://www.oaic.gov.au/privacy/privacy-guidance-for-organisations-and-government-agencies/guidance-on-privacy-and-the-use-of-commercially-available-ai-products