AI Agent Harness: How We Made Norman Faster

An AI assistant can spend a surprising amount of time preparing to do something simple. You ask it to find a transaction. It makes a plan, chooses tools, reviews the result, then asks another model call to write the answer. Each step sounds reasonable. Together, they can turn a quick lookup into a long wait.

We ran into this while adding an agent harness to Norman. We wanted the assistant to keep track of its work and check what had actually been saved before saying it was finished. That was necessary for a product that works with invoices, receipts and transactions. Making every request pass through the same planning and review process added too much overhead.

Our answer was to keep the evidence requirements and make the amount of orchestration depend on the task. During that rollout, we observed substantially shorter completion times. The useful lesson is where we removed work.

What is an AI agent harness?

An agent harness is the software around a model that manages how the agent works: the context it receives, the tools it can use, the state it keeps and the conditions under which a task ends. The model proposes actions. The harness runs those actions and decides which evidence the application needs before accepting completion.

Think about asking an assistant to attach a receipt to a transaction. Understanding the request is one part. Finding the right objects, making the change, checking the saved relationship and reporting an interruption are application responsibilities. A better prompt helps with interpretation. It cannot replace those responsibilities.

The current industry discussion is increasingly about this surrounding software. LangChain's July release of Deep Agents made planning optional and trimmed default instructions. A September research preprint studies retrieving relevant tools instead of presenting an entire catalog. PlayerZero's September analysis argues that harnesses can force models to make unnecessary decisions.

Those are useful questions for a founder building an agent people use during their working day. Every extra decision consumes time, even when the final answer is correct.

Why does a finance agent need a harness?

A polished response is weak evidence that a business task has finished. An assistant might identify several documents, process some of them and produce a convincing summary. The user needs to know which changes actually happened and what remains unresolved.

We added persistent execution state and a journal of tool activity. The application records an intended operation before dispatch and its result afterward. That gives subsequent checks something concrete to inspect. An acknowledgement that a tool ran does not, by itself, establish that the requested outcome exists.

For supported changes, the harness reads the saved object back and compares the relevant fields or relationships. A returned invoice identifier is useful. Finding that invoice with the expected saved values is stronger evidence. This still does not establish that every accounting judgement is correct; checking persistence and checking professional judgement are different problems.

The journal also helps with interrupted operations. When an outcome is uncertain, the next step should resolve the saved state before attempting the same change again. Quietly repeating a write can create another problem while trying to recover from the first.

Why can an agent harness make responses slower?

Our first approach put too much of this control into a general execution loop. Planning made the objective explicit. Checkpoints recorded progress. Review checked the outcome. A final writing step turned the result into a response. Useful building blocks became a costly default for ordinary requests.

The delay was often between business operations. A lookup might return quickly while the assistant continued deciding whether to proceed, reviewing what it already knew or rewriting its own conclusion. Optimizing the database alone would leave those waits in place.

We therefore separated completion time from time to first text, and instrumented setup, generation, tool reads, tool writes and review. Some phases overlap, so adding their durations would give a misleading wall-clock total. We needed to see both the user's total wait and where the agent spent its effort.

This changed the optimization question. Instead of starting with a faster model, we asked which model calls the application was creating unnecessarily. The full execution loop had become one of those costs.

How did we make Norman's agent faster?

We started by choosing the execution path locally. A greeting, a lookup, an ordinary change and a large batch do not need the same control flow. This classification does not require another model call.

For ordinary actions, one agent run can perform the necessary tools and answer. There is no mandatory preliminary plan, outer repair loop or separate answer writer. One run can still contain several model calls as tool results arrive. We removed orchestration around the work without pretending that all work fits into one call.

When a supported change can be checked through exact saved values, code performs that comparison. The model does not need to review it again. More ambiguous outcomes can require a single additional semantic check. That check answers a specific unresolved question rather than restarting the whole task.

We also reduced the context supplied at each step. In a local transaction-lookup comparison, request-specific selection reduced the catalog from roughly sixty tools to fewer than ten. Serialized tool descriptions fell from about 60,000 characters to about 12,000, a reduction of roughly 80%. These are character counts for that comparison, not measured token savings or a universal latency improvement. Our earlier piece on choosing the agent's tool surface explains the broader design principle.

Conversation history now has a bounded budget, while relevant attachments and follow-up references are preserved. Optional external sources connect when needed. Independent overview reads can run concurrently. None of those changes requires caching financial results that may have changed since the last request.

How much faster did Norman become?

Across two complete UTC days during the September rollout, we compared roughly four hundred successful streaming assistant runs. Median completion time moved from about 20 seconds to about 10 seconds. The 90th percentile moved from roughly 90 seconds to roughly 30 seconds. That second measure describes the slower end of successful requests: about nine in ten finished within it.

This was an observation of production traffic on September 8 and 9, not a controlled replay of identical prompts. The request mix changed, several improvements shipped together, and failed or unfinished runs are excluded. We cannot assign the entire difference to one optimization or promise that every task will finish twice as quickly. The numbers describe the rollout sample.

We kept a separate check on perceived speed. Read-only answers can stream as they are generated. Answers claiming a completed change must respect the verification boundary. Showing an early sentence is useful, but it does not make the underlying task complete sooner. Our article on streaming an AI agent covers that distinction at the interface.

The improvement we care about is a shorter route to a useful, supportable result. A quick message followed by a long invisible wait would not satisfy that goal.

How do you test an agent harness?

Tests need to examine the path as well as the answer. Can a normal lookup finish without an unnecessary reviewer? Does a supported write receive its readback? Does an uncertain operation stop or investigate before being repeated? Does the final response describe the evidence the run actually collected?

In scripted checks, a simple lookup and a supported create-and-readback path each used two generation calls, with no separate reviewer or writer. Those checks establish an orchestration budget. They do not measure live-model latency or prove all future responses correct.

We persist scenario results so changes to tools, context or completion logic can be compared later. Production timing complements those scenarios by showing what happens under real traffic. We described the broader observability approach in tracing agents without storing customer content.

I want Norman's harness to spend its effort where uncertainty remains. A large batch deserves explicit progress tracking. A saved value that can be compared exactly deserves a small deterministic check. A simple lookup should get the shortest path that can answer it. That is how we intend to keep useful control as the agent becomes faster.