We do not write test cases for our AI. Production does.

Ask an engineer at any AI company how they know their model got better and you will usually get one of two answers. The confident answer is "we have evals." The honest answer is "we have a spreadsheet somebody made in March."

We have run the honest version. It does not work, and the reason is not laziness. A hand-written golden dataset is a snapshot of the mistakes you could imagine the model making, taken on the day you sat down to imagine them. Our system reads receipts, assigns bookkeeping categories, and drafts tax filings for German businesses. What it gets wrong is decided by which merchants our users pay this quarter, which bank formats send which memo strings, and which corner of German VAT law a subscription lands in. None of that is in my imagination. All of it is in the database.

So we stopped writing test cases and built a pipeline that harvests them.

The signal is the correction, not the complaint

The instinct is to ask users for feedback. Thumbs up, thumbs down, a "was this helpful?" widget. We ship that too, and it is nearly worthless by volume: people do not rate software, they fix it and move on.

The signal with real volume is the fix itself. When our categorizer suggests Software for a transaction and the user quietly changes it to Advertising, that edit is a labelled example with a human in the loop, produced for free, at the exact moment the human cared enough to be precise. Over the last fortnight that path produced several thousand corrections. The explicit thumbs path produced a rounding error.

Two constraints fall out immediately, and both are boring in the way that matters.

First, the recorder cannot live in the request. A user updating a transaction is on a latency-sensitive write path, and nothing about our learning loop is worth one extra second of their time or one extra chance of a 500. The correction is handed to a background task, and that task swallows every exception it produces. If the learning loop breaks, the product does not notice.

# Illustrative sketch, not our source. Shape only.
@deferred(timeout="short")
def on_manual_override(record_ref, machine_value, human_value, actor):
    try:
        local_memory.upsert(record_ref, human_value)
        if machine_value:                 # a machine guessed here, and the guess lost
            defect_log.observe(record_ref, machine_value, human_value)
    except Exception:
        log.warning("override not recorded", exc_info=True)

Second, the loop has to distinguish a wrong answer from a different answer. That turns out to be the whole game.

Most corrections are not defects

A user changing a category is not proof that anything is broken. There are at least four reasons an edit happens, and only one of them is an engineering problem:

• Extraction miss. We read the document or the memo line and got the wrong answer. This is a defect.
• Mapping bug. A deterministic rule fired and mapped to the wrong account. Also a defect, and a more embarrassing one.
• User preference. Two categories are both defensible and this business prefers the other one. Not a defect.
• Tax judgment. The right answer depends on facts only the business or their advisor knows. Not a defect, and emphatically not something to fix in a prompt.

Only the first two become engineering signal. The others are recorded, counted, and firewalled off, because a suite that treats taste as failure will happily optimise your model into somebody else's opinion.

The split between the first two is inferred, not asked. If the transaction already carried a category set by the rule engine, the correction is blamed on the rules; if it carried a model suggestion, it is blamed on extraction. Nobody fills in a form. In production that ratio is lopsided in a way I find reassuring: the deterministic rules account for well under one percent of corrections. When we are wrong, it is almost always the model reading a document, not our own hard-coded mapping.

Fingerprints, not tickets

A raw correction stream is unusable. Thousands of edits are not thousands of problems; they are a few hundred problems with a long tail of noise. So corrections are not stored as work items. They are folded into findings, keyed by a fingerprint.

The fingerprint is a hash over the workflow, the blamed reason, the normalised field path, and an identifier for the counterparty. Same merchant, same field, same failure reason, same fingerprint, and the evidence count goes up by one.

# Illustrative sketch. Identity is derived, never assigned.
def group_key(stage, blame, target, party=None):
    parts = {
        "stage": stage,
        "blame": blame or "unknown",
        "target": canonical(target),
    }
    if party:                    # set only when present, so keys minted
        parts["party"] = party   # before this field existed still match
    return digest(parts)

That if is worth a sentence. When we added the per-counterparty component, the naive version would have re-hashed every historical finding into a new bucket and reset months of accumulated evidence. Adding the key only when it has a value keeps the old three-key fingerprints hashing identically. Schema evolution in a content-addressed store is a database migration wearing a different hat.

Three strikes, then you are a test

Here is the rule I would defend hardest in this whole design:

A finding must recur at least three times before it earns a test case.

The production distribution says exactly why. Of the fingerprints we have accumulated, about two thirds have been seen exactly once. One user, one merchant, one edit, never again. Had all of them become regression cases, the suite would now be mostly noise, every case pinning a behaviour nobody has asked for twice, and every future prompt change would trip a dozen meaningless red lights. A test suite you learn to ignore is worse than no test suite, and the fastest way to build one is to let a firehose write it.

Three strikes turns the volume knob into a filter. Roughly one finding in six clears the bar and is promoted into an eval case, and once a pattern is genuinely recurring it gets there fast: the median time from the first correction to a live regression case is well under a day, fully automatic, with no triage meeting anywhere in the path.

The suite never sees your data

There is an obvious objection to generating tests from production: you have just built a permanent archive of customer bookkeeping inside your CI system.

We do not, and the reason is that the eval cases store hashes of the corrected values, not the values. The nightly replay re-runs the current pipeline over the same inputs, hashes the result, and compares. It can tell you the behaviour changed. It cannot tell you what anybody bought.

The same policy runs one level down. Every unit of model work is recorded as a traced run with the provider, the model, token counts, latency, cost, and a hash of the input and the output. The payloads themselves are never persisted, and each trace row carries the retention policy it was written under, so the guarantee is visible in the data rather than in a document. That constraint is load-bearing, not decorative: it is why nobody has to negotiate about how long the regression corpus may live.

What this looks like at volume

Architecture arguments are cheap without scale, so: over the last two weeks our agent layer traced well over a hundred thousand runs, mostly categorization, peaking north of ten thousand in a single day, with a steady stream of document extraction alongside. That is on the order of a hundred million tokens in each direction.

One number from that pile changed how we write prompts. Around seven in ten input tokens are served from the provider's cached prefix rather than processed fresh, and that is not luck: every prompt is assembled static-part-first, with all per-company and per-user context appended at the end, so the long shared preamble stays byte-identical across users and stays cacheable. It is a rule with teeth. One well-meaning refactor that hoists a company name into the header would quietly torch the cache on every request, which is why our cost model prices cached and uncached input tokens separately. The damage shows up as money.

Where our loop is still open

I would rather write about the part that works, but the interesting engineering is in the seam, and ours had a hole in it.

The nightly replay has been running on schedule for weeks, fifty cases a night, faithfully producing a result row for every one. Every single one of those rows is an error. Not a pass, not a fail: an error, with the same message each time. The half of the system that promotes findings writes an expected outcome in its own vocabulary, describing the reasons and field paths that should stop recurring. The half that grades a replay understands a different vocabulary, the value-comparison kind. Neither half is wrong on its own terms. They were built weeks apart and never made to agree, so the grader reads every case we generate, finds nothing it recognises as an assertion, and gives up.

The loop is complete right up to the last hinge, and then it turns freely. Three lessons, and they generalise past our codebase.

An erroring suite is more dangerous than an absent one. A missing harness is a known gap. A harness that runs nightly, writes rows, and grades nothing looks like coverage on every dashboard. The fix is not more tests, it is treating "graded nothing" as a build-breaking condition instead of a data point.

A self-writing test suite needs a contract test. When the producer and the consumer of test cases are different modules, the format between them is a public interface and deserves its own test. Ours was documented in each half and enforced in neither.

Ambition in the schema is not capability. Every traced run carries a foreign key to the prompt version that produced it, and in production that column is null on all hundred-thousand-plus of them. The registry is real, it is DB-backed, prompts change without a deploy, and the code reads the active version at call time. Then it throws the version away instead of writing it down. So today we cannot answer the one question the apparatus exists to answer: did that prompt change help? Two lines fix it. It survived this long because nothing ever failed, and I now treat "no error, no answer either" as its own category of bug.

The boring parts are the product

None of the interesting work here was model work. The model is a component we rent. The parts we own are the ones that decide which of thousands of human edits deserve to be called defects, that refuse to promote a coincidence into a test, that hold a regression corpus without holding customer data, and that are honest enough to admit when the last link is not connected.

Let production write the test cases, because production knows things you do not. Then be paranoid about the pipeline that carries them, because a learning loop that quietly stops learning will never tell you.