AI Audit Trail: Can You Reconstruct Past Decisions?

An AI assistant explains why a report looked the way it did last quarter. It reads today's records, applies today's rules and produces a plausible answer. The explanation may be useful. It still has not reconstructed the earlier decision.

An AI audit trail needs to distinguish the result we saved, the evidence available at the time and the computation we can reproduce. I would make vendors demonstrate those capabilities separately. A successful rerun against current data is evidence about today, even when the question names an earlier period.

The distinction is becoming a product question. Workiva announced Agent Studio and automated audit testing on September 15, 2026, describing agents that connect evidence collection, testing and documentation. Raza Sharif's September 5 Internet-Draft distinguishes record reproducibility from decision reproducibility. It is an individual proposal, not an adopted IETF standard. Both developments make the same practical question timely: what survives after the system changes?

What should an AI audit trail prove?

Start with the question being asked. “What did we submit?” requires the retained output and evidence of the relevant submission. “Which inputs supported it?” requires the input state or a historical reference that still resolves. “Would the old calculation produce that result again?” also requires the old calculation and its dependencies.

These questions form different contracts. A saved output can establish what a record contains without explaining how it was produced. An explanation can connect inputs to a result without making an entire model execution reproducible. A deterministic calculation can be replayable while the agent that selected it remains outside the replay's scope.

This is why an audit screen should state its evidence level. Someone opening a historical report should not have to infer whether the displayed breakdown is frozen, reconstructed from retained history or calculated from current records. The date of the requested reporting period does not settle that question.

Is a saved result enough to replay a decision?

No. A result is the destination, not the complete route. Think of a synthetic expense that was assigned to one reporting line in June and corrected in July. The saved June total may remain unchanged while the live transaction now belongs elsewhere. Running the current calculator over that transaction answers a different question.

Preserving a transaction identifier alone does not preserve its earlier values. Nor does a content hash restore the content it identifies. A hash can verify retained bytes; it cannot supply missing bytes. This matters when evidence lives in another system whose retention or versioning differs from yours.

The useful buying question is therefore not whether a product “has logs”. Ask which row of this table it can demonstrate for an old case, using retained evidence rather than a newly assembled explanation.

What did we find in Norman's production records?

We checked this distinction in our own production database using aggregate, read-only queries. We found report records marked submitted with a saved payload but no populated dedicated source-allocation snapshot. Saved output payloads were far more common than populated source-allocation snapshots in the records we checked. The query inspected presence, not customer content. It did not inspect the evidence that might exist in other stores.

That finding has a precise boundary. A report status can be set manually, so these records are not a count of verified external submissions. An absent allocation snapshot also does not mean that the saved payload or a separate document is absent. The observation is that preserving an output and preserving its detailed source allocation are different achievements.

Our current source code makes that distinction concrete. For annual VAT reports outside the draft path, the detail reader uses a retained source-allocation snapshot when present. It does not manufacture a historical allocation from today's transactions when that snapshot is missing. This is source inspection, not a claim that we tested every deployed report route.

The lesson is uncomfortable and useful: adding a place to save evidence does not establish that older records contain it. Our earlier AI tax advisor article discusses retaining the submitted allocation. Historical coverage still needs its own check. A feature description cannot substitute for inspecting the records it is supposed to explain.

Which versions must a replay preserve?

For a deterministic calculation, preserve the actual inputs and the rule or code revision that interpreted them. Include configuration that changes the result: applicable period, rounding policy, mapping revision and any external reference data. An identifier that points to a mutable “current” configuration does not freeze that configuration.

For an agent workflow, distinguish the recorded run from a new model execution. Retaining tool responses and selected actions can let you inspect what happened. Asking a model to choose again is a new experiment, even if it reaches the same answer. Our agent memory article makes a related distinction between retained experience and evidence of improved behavior.

Here is an illustrative manifest for a deterministic replay package, not Norman's internal schema:

{
  "scope": "deterministic-calculation",
  "input_snapshot": "retained-input-v1",
  "calculation_revision": "calculator-r7",
  "configuration_snapshot": "retained-config-v3",
  "reference_data_snapshot": "retained-reference-v2",
  "expected_output": "retained-output-v1",
  "external_effects": "disabled"
}

Every reference must resolve to retained material in the review environment. A manifest full of labels is merely an inventory of dependencies. Validate retrieval and integrity before interpreting a mismatch, and record when a required artifact is unavailable. That lets a reviewer distinguish a calculation difference from an experiment that could never run faithfully.

How do you test historical reconstruction safely?

Choose a completed case and define the narrow claim first. For example: the retained inputs and calculator revision reproduce the recorded line totals. Keep the expected output separate from the computation, then run the calculation without external effects. A replay should not send another invoice, payment or filing to discover whether the first one was correct.

Use a deliberate counterexample as well. Change a synthetic input or mapping in a separate copy. The comparison should identify the change, while the historical package remains unchanged. If a supposedly historical replay silently reads the modified live value, the test has exposed its dependency on current state.

Keep recovery testing separate. Our article on agent retries asks where an interrupted run can resume. Historical reconstruction asks which earlier state remains available after the run is over. Passing a retry test proves neither long-term evidence retention nor the ability to reproduce an old calculation.

A failed reconstruction should produce a useful diagnosis: missing input, unavailable revision, changed result or unsupported replay scope. These are different outcomes with different remedies. Treating them all as “AI was wrong” conceals the engineering work needed to improve the system.

What should you ask an AI vendor to demonstrate?

Ask for a historical case after its live inputs have changed. Have the vendor identify the retained output, show the provenance of the breakdown and explain which computation can run again. Then ask what happens for records created before the current evidence mechanism existed.

The strongest demonstration is bounded. It states which artifacts are available, which comparison passed and which part still depends on interpretation. It also has an explicit answer for missing history. “We can show the saved result, but cannot reproduce this earlier allocation” is useful information for someone deciding what to review next.

I would accept a smaller reproducibility promise that survives that demonstration. The promise that matters is not that an assistant can always generate an explanation. It is that the product can tell a reviewer which parts of that explanation are supported by retained evidence, after the easy path of consulting today's state is taken away.