Why check at all
An AI-written analysis reads fluently whether or not the numbers are right. A figure with no denominator, or one that cannot be traced to a record, should not reach a board, a regulator or a staff member's file. These twelve checks work on any AI-generated compliance analysis, whoever produced it. Each lists the question to ask and what a bad answer looks like. Ask all twelve before you rely on the document.
Is the denominator stated?
Ask: "Out of what total is this number?"
Bad answer: "47 flagged" with no count of what was screened.
Can every figure be traced to a record?
Ask: "Show me the rows behind this number."
Bad answer: A summary that cannot list the underlying records, or a list that does not add up to the figure.
Are the model and prompt versions recorded?
Ask: "Which model, which version, which prompt, on what date?"
Bad answer: "We used AI." Without versions, nobody can reproduce the result.
Are missing fields excluded or counted as zero?
Ask: "How many records had a blank in each field used, and what happened to them?"
Bad answer: No answer, or blanks silently treated as zero, which pulls averages down and makes quiet staff look safe.
Is a null result reported as a null?
Ask: "What did the analysis find where it found nothing?"
Bad answer: A pattern presented from a handful of cases, or a "trend" that is not larger than the noise.
Who signs off?
Ask: "Which named person reviewed this and what did they check?"
Bad answer: No name, or a name with no record of what was reviewed.
Was the data period and extraction date stated?
Ask: "What dates does this cover, and when was the data pulled?"
Bad answer: "Recent data."
Were the exclusions listed?
Ask: "What was left out, and why?"
Bad answer: No exclusions listed, which almost never means none.
Do the totals reconcile?
Ask: "Do the parts sum to the whole, and does the whole match the source system's count?"
Bad answer: Percentages that sum to 104, or counts that differ from the pharmacy system's own report.
Is a finding separated from an inference?
Ask: "Which statements are counted from data and which are the model's interpretation?"
Bad answer: "Suggests intent" or "indicates diversion" written as though counted.
Was a human spot-check done on a sample?
Ask: "How many records did a person verify by hand, and how many were wrong?"
Bad answer: No sample, or a sample taken only from the records the model flagged.
Are the limits of the source data stated?
Ask: "What can this data not tell us?"
Bad answer: No limitations section, or a generic sentence that could appear in any document.
A worked example
Suppose an AI summary says: "Diversion cases rose sharply in 2026." Run the checks. Check 1: out of what? The DivertGuard registry holds 127 cases dated 2025 and 93 dated 2026, but the 2026 figure runs only through 30 October, so a fair comparison needs the same months in both years. Check 4: a month-only date is not the same as a missing date, and the registry has 351 month-only dates out of 931 records, so any month-level count needs a stated rule for them. Check 12: the source is published cases, which lag the events behind them. The honest sentence is: "Cases published so far in 2026 (93, through 30 October) cannot yet be compared with 2025 (127)." The AI summary skipped all three.
Where AI analyses usually fail
Three patterns recur. The model fills a gap with a plausible number. It writes a confident sentence about a pattern it was never shown. And it silently changes a definition between paragraphs, so "case" means an alert in one place and a confirmed finding in another. Checks 2, 8 and 10 catch these.
How to use the list
Count a check as failed when the answer is missing, not only when it is wrong. One failed check among 1, 2, 4 or 6 means the analysis is not ready for an inspector, a board or an employment decision. Failures elsewhere mean it goes back for revision. Keep the completed checklist with the analysis, with the reviewer's name and date.
Apply it to your own work as well. Several of the registry's published figures on DivertGuard state their denominator beside the number for this reason; see how each figure was counted.