How we trace every AI agent run without storing your data

Every AI feature we ship runs a model against something private. Categorization reads a bank transaction. OCR reads a receipt. Reconciliation compares an invoice to a payment. Autofiling assembles a whole VAT return. If you want to run agents in production and still sleep at night, you need to be able to answer, months later, "what exactly did this run do, and why?" The naive way to get that answer is to log everything: every prompt, every response, every document the model looked at.

We do the opposite. We trace tens of thousands of agent runs, and we deliberately keep almost none of the data. What we store is the shape of a run, its cost, its provider, and a cryptographic fingerprint of the input and output. The receipts and the amounts stay in the tables they belong to. This post is about that trade, and the small pile of boring machinery that makes it hold.

One run, one row

Underneath every AI feature sits a single app whose only job is to make model work legible. Every unit of that work becomes one row, an agent run. The interesting columns are not the payload, because there isn't one. Roughly, a run looks like this:

AgentRun
  workflow            ocr | categorization | reconciliation | autofiling | ...
  status              running | needs_input | succeeded | failed | ...
  provider, model     who ran it
  input_hash          fingerprint of what went in
  output_hash         fingerprint of what came out
  estimated_cost      in micro-USD
  latency_ms
  prompt_version      which prompt produced it

A run has children. Each step is one model call or tool call inside it, with its own token counts and cost. Each artifact is a structured output worth keeping (the fields OCR extracted, the answer the copilot returned). Call sites do not build any of this by hand. They wrap the model call in a context manager and set the result on the way out, roughly:

with traced_agent_run(workflow="categorization", input_payload=payload) as trace:
    result = call_model(payload)
    trace.set_output(result, response=response)

That is the entire contract. Start a run, do the work, hand back the output. The layer records the step, rolls up the cost and latency, closes the run, and writes an artifact if you asked for one. Six real workflows in production go through this exact path today.

We store the hash, not the data

Look again at input_payload above. We pass the model's actual input into the tracer, and yet the row only ever holds an input_hash. That is the whole idea. Before anything is written, the payload is reduced to a fingerprint, roughly:

input_hash = sha256(canonical_json(payload))

The word that carries the weight is canonical: we serialize with sorted keys, so the same logical input always produces the same bytes and therefore the same fingerprint, regardless of dict ordering. A 64-character digest replaces a receipt, a counterparty name, or a full transaction list.

A hash cannot be read back into a document, which is exactly what we want. But it is not useless. It lets us answer the questions that actually come up in production:

• Did the input change? If a run is re-triggered and the input_hash matches, nothing upstream moved. If it differs, something did. That is the basis for idempotency and for detecting silent drift.
• Did the model give the same answer twice? Equal output_hash values mean identical output, without diffing two blobs of customer data.
• Which correction belongs to which run? When a user fixes a categorization, we can bind the fix to the exact run that produced it by fingerprint, and never have to store what the model saw to do it.

Every trace row also carries its own policy label, so the intent is self-documenting and enforced in one place: a retention policy of "metadata and hashes only," and a payload policy that says inputs and outputs are hashed rather than stored. Artifacts are the one exception, and a deliberate one. When we do keep a structured output, it is a small, known-shape object we produced (extracted fields, a short summary), never the raw source document, and it is hashed too. The raw receipt already lives in its own table with its own access controls. The trace does not get a second copy.

Tracing must never break the thing it traces

An observability layer that can crash the feature it observes is worse than no layer at all. So the entire tracer is built on one rule: a tracing failure is invisible to the caller. It shows up in three places.

First, the layer can be switched off wholesale, and every entry point checks that flag before touching the database. Second, every write is defensive: a failure to record is caught, logged, and turned into a "no trace" result instead of an exception, so a bad write to the trace table can never propagate up into categorization. Third, every downstream helper treats a missing run as "tracing is not happening" and quietly does nothing.

The one thing the tracer does not swallow is your error. If the model call inside the block throws, the run is recorded as failed and the exception is re-raised to you unchanged. We hide our own failures, never yours. The result is that categorization keeps working whether or not the trace table is having a good day, which is the only acceptable behavior for something that wraps every model call in the product.

Cost is a first-class column, not a monthly surprise

Because the layer sees every model call, it is the natural place to count what they cost. Each step records input, output, and cached token counts, and turns them into an estimated cost in micro-USD, not dollars: a single categorization can cost a tiny fraction of a cent, and integer micro-dollars let us sum millions of them without floating-point rounding drift.

Pricing is not hardcoded. A per-model policy row holds the per-1k-token prices, a latency ceiling, a risk level, and an optional cost budget, so changing what a model costs us is a data change, not a deploy. When steps roll up into their run, the running totals for cost and latency are written with a single atomic increment rather than a read-modify-write, so concurrent steps on the same run cannot clobber each other's numbers. If a run blows past its policy's budget, we tag it rather than killing it mid-flight, so the anomaly is queryable after the fact. On top of this we compute unit costs that a finance person actually cares about: cost per thousand categorized transactions, cost per OCR document, cost per submitted VAT return. When "how much does the AI cost us" comes up, the answer is a query, not a guess. At our current volume, that query runs over close to a million transactions on our customers' books.

Prompts are data, not deploys

There is one more thing the run points at: the exact prompt version that produced it. Prompts live in the database, keyed by workflow and version, and the active one is resolved at call time behind a short cache. The safety property is a small one but it matters: an empty override means "fall back to the prompt baked into the code," so the worst case of a misconfigured prompt row is that we quietly use the version that shipped. The code is always the safe default; activating or deactivating a row can never break the pipeline.

That buys two things. You can iterate on wording without a release. And because every run is stamped with the version that made it, you can later tell which prompt produced which output. That stamping is the thing that turns "we changed the prompt and it feels better" into something you can actually measure.

The payoff: an agent you can grade

All of this exists to support one goal, which is making the agent measurably better without letting it change itself. The same run that carries a hash and a prompt version also collects human signal: a like or dislike, and structured corrections when a user overrides what the model produced. We keep a few hundred of those corrections as typed review rows, each tagged with a reason (extraction miss, mapping bug, unsupported case, plain user preference), so a pattern of real mistakes can be told apart from noise and frozen into an evaluation case.

That eval loop is young. We have the plumbing (findings, eval cases, graded results) and are still filling the dataset, so I will not pretend we run a mature regression suite yet. But the substrate is the point. Because every run is fingerprinted, versioned, and costed, the moment a correction is worth learning from, we can turn it into a repeatable test and prove the next prompt does better, with a person, never the model, deciding what ships.

Boring on purpose

None of this is the exciting part of an AI product. The exciting part is the model reading a receipt and getting it right. But the model getting it right in a demo and the model getting it right on someone's tax filing are different claims, and the distance between them is exactly this kind of unglamorous bookkeeping: a row per run, a hash instead of a payload, a cost you can query, a prompt you can name, and a tracer that would rather disappear than take the feature down with it. Automate the boring work, keep the human on the decision that carries risk, and be able to prove every step in between. The observability layer is how we keep the last promise.