In partnership with
What if AI made us prove we’re creative?
This year AI shook up creative strategy. Most teams are still working like it didn't.
Ari Murray, Chief Digital Officer at Salt and Stone sits down with Bryan Harvey, Head of Brand at Hightouch, for an honest conversation about what's breaking on creative teams right now, how the roles are changing, and what a day looks like when creatives spend it on ideas, concepts, and craft while agents handle the manual execution.
Leave with new insights from 3 experts in the field and 5 prompts to instantly improve your creative output.
iPrompt
DEEP DIVE · iPROMPT #157 COMPANION
14 OCTOBER 2026
How to verify AI output
Write the check before you ask
By the end you’ll have a check for one recurring AI task, tested against mistakes you planted yourself, and a clear idea of what passing it does and doesn’t prove.
6 MIN READ · WORKED EXAMPLE AND CHECKLIST
VERIFYING AI OUTPUT / WORKED EXAMPLE
One quarter’s invoices, checked by design
Kestrel Home is fictional, and so are its figures. It’s a 15-person online homeware shop whose bookkeeper codes about 1,200 supplier invoice lines a quarter to ledger accounts and VAT rates. This quarter an AI assistant did the coding. Before it ran, the bookkeeper wrote down five checks and a stop rule: fewer than 38 of 40 known answers right, and nothing goes to the ledger. To test the checks, a colleague planted four errors in a copy of each run’s output.
Check | Time per quarter | First run | Re-run |
Control total: coded lines sum to the invoice register | Seconds | £1,214.40 short: two lines on one invoice merged, one amount lost | Reconciled to the penny |
VAT rule: rate matches supplier and product type, every line | 2 minutes | Nine books coded at 20%; UK books are zero-rated | Two books at 20%, corrected |
Duplicates: same supplier, amount and date within 7 days | 2 minutes | One supplier had billed twice | None found |
Known answers: 40 past lines, mixed in unlabelled | 10 minutes | 36 of 40. All four misses were packaging coded as stock. Stopped | 39 of 40 on a fresh set. One credit note coded as a purchase |
Human sample: 30 random lines | 25 minutes | One judgement call, no errors | No errors |
Packaging rule (added after the first run) | 2 minutes | Not yet in place | None found |
Planted errors: four per run, set by a colleague | 10 minutes | 3 of 4 caught. A packaging line coded as stock got through | 4 of 4 caught, a new packaging line among them |
Closing the loop. The first run broke the stop rule, so it never reached the ledger. Two changes followed: a line in the prompt naming the packaging accounts, and a rule that flags any line from Kestrel’s six packaging suppliers not coded to packaging. The re-run was scored against 40 fresh known answers, because the first 40 had been used to fix the prompt and could no longer test it fairly. The one miss was corrected by hand and added to the pool.
What the pass proves. Less than it looks. 39 of 40 clears the stop rule, but one miss in 40 fits a true error rate anywhere from under 0.1% to about 13% (a standard 95% interval). The sample can’t close that gap: even if four of 1,200 lines were wrong, a simple random sample of 30 would have about a 9.6% chance of including one. Rare errors need a rule, not a sample. So the checks that cover every line (totals, VAT, duplicates, packaging) are firm, and account coding elsewhere is only sampled. Approving is a decision to accept that leftover risk, not proof there isn’t any. Kestrel accepted it because VAT, the part HMRC sees, was checked on every line.
The record. ‘Q3 invoice coding, 14 October. 1,206 lines. Run 1 stopped: known answers 36/40, planted errors 3/4. Prompt fixed; packaging rule added. Run 2: totals reconciled; two VAT rates corrected; fresh known answers 39/40, credit-note miss corrected; planted errors 4/4. Approved by the bookkeeper, accepting that some account coding outside the rules may be wrong.’
Setting up took about 90 minutes, mostly choosing known answers. Each quarter from now on takes about 50, half of it the human sample.
VERIFYING AI OUTPUT / THE MATHS PARALLEL
What a machine check proves
OpenAI’s 6 October maths release has the same shape. Its README says about 42% of top-line results are formalised in Lean, which verifies every step, but only of the statement that was formalised, not every claim in the paper. Kestrel’s control total works the same way: it proves the sums, not the accounts.
VERIFYING AI OUTPUT / THE SETUP
Five steps for one recurring task
This is the step before last week’s companion on reviewing AI decisions. That piece checks an answer you’ve been given; this one decides in advance what a right answer has to pass.
1 Choose. Pick a task you’ll repeat at least monthly, where a wrong output costs money or trust. A one-off is cheaper to review by hand.
2 Write the checks first. Run the newsletter’s prompt before the task prompt. Keep the checks a formula can run, then the few that need a person. Write them down before you see any output. The AI can draft them; you choose them. Drop any check you couldn’t run or explain yourself.
3 Build known answers. Take 20 to 40 items you’ve already handled correctly, awkward cases included, and mix them in unlabelled. Set the stop rule now: the score below which the output isn’t used. After you change the prompt, re-test on a fresh set.
4 Plant errors. In a copy of the output, plant three or four realistic mistakes, ideally chosen by someone else. If one gets through, add or change a check, then plant fresh errors to confirm the change works.
5 Record and repeat. One line per run: what was checked, what failed, what changed, who approved. Plant fresh errors each quarter, and whenever the model or the prompt changes.
Checks, cheapest first
Check | Example | Catches | Misses |
Reconciliation | Coded lines sum to the register | Dropped, merged or inflated lines | Right total, wrong account |
Rules | VAT rate matches product type | Values outside what’s allowed | Errors that follow the rules |
Known answers | 40 lines from last quarter | Repeated misreadings | Rare cases not in the set |
Planted errors | A duplicate line in a copy | Gaps in your other checks | Mistakes nobody thought to plant |
Human sample | 30 random lines | Judgement calls | Rare errors, most of the time |
The strongest objection
‘Writing checks takes longer than checking the output.’ For a one-off, it does, so just review it. Checks pay off on work that repeats: Kestrel’s take about 50 minutes a quarter once set up, while reading 1,200 lines doesn’t get any quicker. Undo’s survey of 300 senior engineering leaders at large US and UK firms shows the other path: 79% said releases are no faster with AI, and their engineers spend more of the week debugging than writing code.
THIS WEEK’S FORECAST
By 30 June 2027, the README of OpenAI’s maths repository will report that at least 75% of top-line results are formalised in Lean. On 14 October 2026 it said about 42%.
Why 75%. OpenAI says it will add formalisations ‘as we obtain them’, and formalising is work AI models increasingly do themselves. The hardest results will be slowest, so I don’t expect all of them.
Counting. It counts if the README, or an OpenAI page it links to, gives 75% or more by the deadline, however many results there are by then: additions and withdrawals count as the README reports them. Formalisations by others count once the README includes them. Void if the repository is deleted or stops giving a figure; a miss if the last figure before the deadline is below 75%.
YOUR MOVE
Set up checks for one recurring task with the five steps, and plant at least one error. Reply with one line: the task, and whether the planted error got through. Leave out names and confidential details.
The aim isn’t to trust the output. It’s to know, quickly, when not to.
New to iPrompt? Every Wednesday, the week’s AI news turned into one thing you can do. Subscribe free
iPROMPT / 157 /