How our categorizer learns from a single correction

Every bank transaction on Norman needs a category before it can become bookkeeping. That is not a nice-to-have: the category decides the tax treatment, whether input VAT can be reclaimed, which line of the Umsatzsteuervoranmeldung the amount lands on. Get it wrong and the return is wrong. At the scale we run this, around a million transactions categorized so far and a six-figure count of new ones landing every month, nobody is going to hand-sort them. So the software has to.

The obvious way to build this in 2026 is to point an LLM at the description and ask. That works, and it is also the least interesting part. A model that reads "Kartenzahlung STEAM PURCHASE Berlin" and guesses Software is table stakes. The engineering problem is the one underneath: when a user overrules the machine and picks a different category by hand, that correction is the single best training signal we will ever get for that business. The whole design is about capturing it, and never wasting it.

The cheapest categorizer is the one you never call

An LLM call is the slowest, priciest, least predictable option we have, so it is the last thing we reach for, not the first. Categorization is a cascade, and each tier only runs if the one above it came up empty:

def categorize(self) -> CategorizationResult:
    result = self.categorize_by_accounting_rules()
    if result.category or result.company_category:
        return result

    result = self.categorize_by_company_pattern()
    if result.category or result.company_category:
        return result

    return self.categorize_by_ai()

First the deterministic accounting rule engine: well over a thousand rules, curated and per-company, matching on counterparty, IBAN, amount sign, and the like. If a rule fires, we are done, no model, no latency. Then company memory, which is the subject of this post. Only if both miss do we spend an LLM call. Nine in ten transactions in production already carry a category, and the ones the model has to reason about from scratch are a shrinking minority. Every correction a user makes pushes another counterparty out of the "ask the model" bucket and into the "we already know this" bucket, permanently.

A correction is the strongest signal we have

The capture point is boring on purpose. When a PATCH changes a transaction's category, the updater notices the field moved, grabs the previous values before they are overwritten, and schedules the learning work for after the database commit:

def capture_categorization_correction(self) -> None:
    # Implicit feedback for the learning loop: the user hand-picking a
    # category is the strongest training signal we have. Captured before
    # the setattr pass (we need the previous values), dispatched after
    # commit so a failed PATCH never records anything.

Two invariants live in that comment. We read the old category before the write, because the correction is the delta between what the machine said and what the human chose, and once the row is updated the "before" is gone. And we dispatch on on_commit, so a PATCH that rolls back for any reason never teaches the system a lesson that did not actually happen. No correction is invented; none is lost.

Memory that has to earn trust

The memory itself is a per-company table, CompanyPattern. The docstring states the one rule that matters most:

class CompanyPattern(TimestampedModel):
    """Company-scoped memory for agents.

    Grown from implicit feedback (user corrections) and consumed by the
    categorizer and the assistant: "for THIS company, this counterparty is
    Software", etc. Deliberately per-company: one company's corrections must
    never leak into another's suggestions.
    """

The key is not the raw bank description, which is full of per-payment noise: card ids, dates, reference numbers. We strip the digits and keep the first few stable tokens, so Kartenzahlung STEAM PURCHASE 12345 Berlin and next month's Kartenzahlung STEAM PURCHASE 67890 Berlin normalize to the same counterparty key.

Crucially, one correction is not enough to act on automatically. A remembered pattern only books a transaction on its own once it has been confirmed at least twice:

# A pattern must be confirmed at least twice before it applies deterministically.
CONFIRMED_EVIDENCE_THRESHOLD = 2

And when a later correction conflicts with what we remembered, the newest human decision wins, but it has to start over earning trust:

if pattern.value == value:
    pattern.evidence_count += 1
else:
    pattern.metadata = {**pattern.metadata, "previousValue": pattern.value}
    pattern.value = value
    pattern.evidence_count = 1

A repeated identical correction is a confirmation: evidence grows, and once it crosses the threshold the counterparty books instantly with no model call at all. A contradicting correction resets the evidence to one and stashes the old value in metadata. The latest decision is always respected, but it never gets to skip the model until the business has shown, more than once, that it really means it.

The same memory, used two ways

Here is the part I like: the exact same table feeds two very different consumers.

Below the threshold, or when the cashflow direction does not match, the pattern is not trusted to decide on its own. But it is still the best hint we have, so it rides into the LLM prompt as a few-shot block, describing how this business books its recurring counterparties:

def company_pattern_examples(company, *, cashflow_type=""):
    """Few-shot block for the categorization prompt: how THIS company books
    its recurring counterparties. Names only, never ids."""

The output reads like 'steam purchase' -> Software; 'aws' -> IT services. Names only, never database ids, because the model has no business seeing our primary keys and we have no business trusting it to echo them back. Above the threshold, the same pattern short-circuits the whole thing and books deterministically. One memory, two speeds: a confident hit skips the model entirely, an uncertain one biases the model toward this company's own history.

One company's memory is never another's

Per-company isolation is not a comment, it is a uniqueness constraint on (company, kind, key) and a company= filter on every read. A freelancer who books Steam as a business expense and another who never would are two separate memories that never see each other.

One more guard earns its place here. Freelancer bookkeeping in Germany assigns only leaf categories, never a top-level parent. So even a remembered, twice-confirmed pattern is refused if it points at a parent category:

# Freelancer bookkeeping assigns only child categories: a remembered
# top-level parent must never be applied. SME companies use their own
# flat chart-of-accounts set and are exempt.
if category and category.parent_id is None and not company.is_sme:
    category = None

The memory is allowed to be wrong in the direction of "ask again," never in the direction of "book something the tax logic downstream cannot handle."

Proving a fix actually stuck

Growing a memory is easy. Proving that a correction stopped the mistake from recurring is the hard, boring, valuable part. When a user overrules a category the machine had already set, we do not just update the memory; we mint an eval case from the correction, keyed by a fingerprint of the miss. A nightly job replays those cases through the current pipeline and checks whether the machine now produces what the human chose:

def categorization_replay_worker(input_payload):
    """Replay the eval case's source transactions through the CURRENT
    categorization pipeline (rules -> company patterns -> LLM) and report
    whether it now produces the user-corrected values.

    Eval cases are redacted (hashes, no raw values), so the replay resolves
    the source transactions by public id and compares stable hashes of the
    computed correction shape against the recorded ones.
    """

Two details make this safe to keep around forever. The eval cases are redacted: they store hashes of the corrected shape, not raw categories or descriptions, so the regression suite carries no customer content. And the check is a stable-hash comparison, computed_hash == recorded_hash, which gives a crisp shouldNotRecur boolean per finding instead of a fuzzy score somebody has to interpret. A fix that regresses lights up the next morning.

Prompts are data, not deploys

The categorization prompt is not a string frozen into a release. It is a row:

def get_active_prompt(workflow: str) -> tuple[str, str]:
    """Prompt-as-data: an empty body means "use the hardcoded default": the
    code is always a safe fallback, so activating/deactivating rows in the
    admin can never break the pipeline."""

An empty override means "use the default baked into the code," so the worst a bad prompt row can do is fall back to the version we shipped. The prompt can change without a deploy, every run records exactly which prompt version produced it, and the nightly replay tells us whether the change actually helped before anyone commits to it.

The boring parts are the product

There is a flashier version of this feature: a chat box where you tell an assistant "book everything from Steam as Software" and it does. We did not build that, because the value is not in the moment of instruction, it is in never having to give the instruction twice. So the design leans entirely on the unglamorous machinery. A cascade that avoids the model whenever it can. A memory that has to be confirmed before it is trusted, and re-earns trust when it is wrong. Strict per-company isolation enforced by a constraint, not a promise. And an eval loop that can prove, the next morning, that your correction actually took. Automate the tedious work completely, keep every guard explicit, and be able to show that the system learned exactly what you taught it, and nothing you did not.