AI Transaction Matching: Scores Are Not Confidence

You open a receipt and the software puts a payment at the top of the list. The amount looks close. The merchant words look familiar. Approving it would take one click. What, exactly, has the system established?

AI transaction matching needs an answer more precise than “the score is high.” A ranked suggestion can save time without establishing that the receipt belongs to that payment. I would rather ship a useful shortlist with an explicit decision boundary than turn an ordering rule into an unexplained confidence percentage.

The distinction matters as reconciliation becomes more autonomous. On September 23, 2026, Microsoft described bulk exception resolution in its Dynamics 365 reconciliation roadmap, which mixes available and forthcoming capabilities. Source: Microsoft. The same day, Smartstream announced autonomous investigation and resolution of reconciliation exceptions with logged actions. Source: Smartstream.

My question for such systems is specific: what permits a suggested match to become a recorded relationship? Here is a small Norman demo that makes that question visible.

What does AI transaction matching actually score?

Inside an AI accounting product, some useful decisions still begin with ordinary arithmetic. Norman's document review ranker combines amount similarity, date proximity and overlapping words. This particular function is deterministic. The replay below makes no model call and performs no bank operation.

We ran the actual product function locally against fictional records. The receipt is for €120 from “Demo Office,” with “paper supplies” in its description. Both candidate payments have the same date as the receipt. Payment A is €120; payment B is €122. Initially, their descriptions are generic.

The resulting order puts A first. That tells the reviewer where to look. It does not identify who bought the supplies or explain a difference between a receipt and a payment. Our broader AI bank reconciliation article describes the exception workflow; this demo isolates the narrower decision of how to order candidates.

Can changing text change the best match?

Now keep the amounts, dates and candidate set fixed. Change only B's description to “Demo Office paper supplies.” B moves above A. Give A the same description and A returns to first place. The animation uses the computed replay results, with internal points exposed for explanation.

Every amount, label and record in this demo is fictional. It is an annotated local replay of product logic, not a screen recording of a customer's account. The real review screen does not display these point totals or a confidence percentage.

Neither move proves that the newly leading candidate is correct. We supplied no verified receipt-to-payment relationship. The experiment proves that text can outweigh the difference between these amount matches under the current rule. That is a useful, reproducible statement with a much smaller scope than “AI matched the payment.”

Why is a matching score not a probability?

The points are contributions to a sorting rule. In this example, a result can exceed 100. Calling it a percentage would therefore be plainly wrong. Dividing every result by the largest possible score would keep the order, but would not create evidence about how often the first candidate is correct.

A probability claim needs outcomes. For example, take a held-out set of verified relationships, group predictions by the confidence they report and compare those predictions with observed correctness. That is an evaluation proposal, not a benchmark we have run for this article.

The distinction also changes how explanations should read. “The amount contributed to this suggestion” is weaker than “the amount is identical.” In our replay, the nearby amount contributes points too. Reason labels help someone investigate, but they should not silently become claims of equality. We make the same separation between a recommendation and a justified action in our discussion of AI pre-accounting.

What happens when two payments look the same?

We also replayed two candidates with identical amounts and dates, generic descriptions and equal scores. The rule used description ordering to break the tie. Alphabetical order is a sensible way to make a list stable. It is not evidence about which payment belongs to the receipt.

This is not purely a toy concern. A read-only aggregate check of Norman's production imported expense records found repeated combinations of absolute amount and UTC value-date within the same company and exchanged currency. Among records not marked as deleted, with company, amount and currency present, roughly one in ten in the checked cohort belonged to a repeated combination. We checked a ninety-day value-date window ending before September 24. No descriptions or individual records were retrieved.

That result establishes that amount and date do not uniquely identify a stored record in that cohort. It does not establish duplicate payments, incorrect links, how often reviewers encounter ambiguity, or why those records exist. The production check and the synthetic replay answer different questions. Together, they justify testing ambiguity without presenting a manufactured failure as a customer incident.

What does a person confirm in Norman?

The reviewed Norman interface shows the payment description, date, amount and matching reasons. Selection is separate from ordering. A reviewer chooses a candidate and then confirms the link. Moving to the next document resets that selection.

Those details matter more than a confidence badge. The first row is where the software directs attention; the selected row is the relationship the reviewer intends to create. Keeping those states separate makes it possible to disagree with the ranking without fighting the interface.

There is another boundary worth stating: this frontend shortlist is not Norman's separate automatic document matching service. The replay does not test that service, a live linking request or a successful accounting outcome. It demonstrates one ranking function and the review interaction supported by the current source. Also, a shortlist only ranks the candidates supplied to it. Missing evidence elsewhere cannot be rescued by sorting the visible list more confidently.

How should an agent use ranked matches?

An agent consuming a shortlist needs the same distinction as a person. Its input should preserve the candidate identity, the signals behind the ranking and any unresolved discrepancy. A nearby amount should remain visibly nearby even when the candidate appears first.

I would design the action contract around the requested relationship: which document, which payment, what supporting evidence, and which rule authorizes the link. A policy may permit automatic handling for a verified class of cases. That permission should be explicit and evaluated separately from the retrieval ranking.

This is a design recommendation, not a claim that our replay has implemented an agent approval system. Our article on AI agent tool results explains why the meaning of returned data matters. Here the crucial meaning is modest: “consider these candidates” gives an agent a place to begin, while leaving the final decision open.

What should you test before automating reconciliation?

Start with cases that change one thing at a time. Keep money and dates constant while varying the description. Then remove the supporting words, add a same-amount competitor, or leave the true payment outside the candidate set. Record both the order and whether the system proposes an action.

For the last case, an empty or unresolved answer may be the desired result. A ranking algorithm will still produce a first row when every available candidate is wrong. Evaluation should therefore include the ability to stop, not just the ability to put a known match first.

Include incomplete records too. A missing date is not the same as a distant date, and an empty description is not evidence that two merchants differ. Write the expected behavior before running each case: should the candidate remain visible, should the system ask for information, or should it decline to link? That makes the test reviewable. Otherwise, a demonstration can look successful merely because it always produces a neat list, regardless of what the available information actually supports.

This demo is small by design. It lets a reviewer inspect why a result moves without needing to trust a dramatic success rate. Before granting an agent permission to link records, I want evidence that it can distinguish a useful suggestion from sufficient evidence. The top row alone cannot provide that distinction.