Streaming an AI agent is a protocol problem

The demo version of an AI chat feature takes about an afternoon. You open a streaming response, you forward tokens, the text appears letter by letter, everyone is delighted. Then you put it in front of real users doing real work, and you learn that the streaming part was never the feature. The feature is a protocol, and almost everything interesting in it is about what happens when the happy path does not happen.

Norman's in-app assistant is a tool-using agent. It can look at your transactions, draft an invoice, prepare a bill payment, explain a VAT number. Across our production traffic we have logged tens of thousands of chat turns, and close to a hundred thousand traced agent runs across all of our AI workflows. The chat turns are the slowest and the most visible of them, and they have a shape that a naive token stream handles badly:

• The median turn finishes in roughly two seconds.
• The 90th percentile is around fifteen seconds.
• The 99th percentile is around thirty seconds, and the worst turns we have recorded run past a minute.

Roughly one reply in five calls at least one tool. Those are exactly the slow ones, and they are the ones where the user is most invested in the answer, because they asked the assistant to do something rather than explain something.

That distribution is the whole design brief. A protocol that is comfortable at two seconds and falls apart at thirty is not good enough, because the thirty second turns are the ones that matter.

The transport is deliberately boring

Our backend is a synchronous Django application. The agent loop is asynchronous. Those two facts do not want to be friends, and there is a well known temptation to rewrite half of the stack to make them friends.

We did not. The streaming view spawns the async agent loop in a daemon thread, and that loop pushes finished lines into an ordinary thread-safe queue. The request generator, running in the normal synchronous worker, drains the queue and yields each line to the HTTP response until it sees a sentinel:

def generate_streaming_content():
    q: sync_queue.Queue = sync_queue.Queue()

    async def run_async() -> None:
        try:
            async for chunk in wrapper.process_assistant_by_prompt_streamed(...):
                q.put(chunk + "\n")
        except Exception:
            logger.exception("Streaming view error")
            q.put(json.dumps({"type": "error", "message": "..."}) + "\n")
        finally:
            q.put(None)  # sentinel

    threading.Thread(target=lambda: asyncio.run(run_async()), daemon=True).start()

The payload is newline delimited JSON. One event per line, no framing cleverness, no partial objects. The client keeps the last incomplete line in a buffer and parses only whole ones. That is four lines of client code and it is the only framing rule in the entire protocol.

I like this layer precisely because there is nothing to say about it. It has never been the thing that broke.

Every event type is additive

The stream carries a small set of typed events: text deltas, a tool start, an image, an action card, a completion, an error, and a trailing save confirmation. The client dispatches on type with a plain if-chain and silently ignores anything it does not recognise.

That single property, unknown events are skipped rather than fatal, is what lets us evolve the protocol without a coordinated release. Web, iOS and Android clients all consume this stream, and they update on completely different schedules. If a new event type were a parse error, every improvement to the assistant would turn into a release train.

Both of the events we shipped this week are proof that it works. Neither required a client change to be safe to deploy, only a client change to be useful.

Silence is the failure mode, not slowness

Here is the thing nobody warns you about when you add tools to a chat agent: a tool call streams nothing. The model decides to call a tool, and then the turn goes completely quiet for as long as that tool takes. No tokens, no progress, nothing on the wire.

For those tens of seconds our UI showed a typing indicator. A typing indicator is a lie in that moment. The model is not typing, it is reading three months of transactions, and the user has no way to distinguish "working hard" from "hung".

The fix is not a spinner improvement. It is an event. The streaming loop already knew the tool name, because it uses it to map tool output onto action cards, so we started sending it:

if event.item.type == "tool_call_item":
    self.current_tool_id = _tool_name_of(event.item.raw_item)
    # A turn can spend tens of seconds inside tool calls with nothing
    # streamed, so without this the UI just shows a typing indicator
    # and the user cannot tell whether anything is happening.
    if self.current_tool_id:
        yield json.dumps({"type": "tool_start", "tool_name": self.current_tool_id})

Now the client can say "Creating the invoice" instead of showing three animated dots. Same latency, completely different experience. A slow operation that tells you what it is doing is tolerable. A fast one that says nothing is not.

The id arrives after the text, on purpose

Our chat replies have like and dislike buttons. Reactions address a message by its persisted id.

A streamed reply does not have one. The database row is only written once the stream is finished, because until then there is no final text to write. So the client rendered a message with no id, and the only replies you could actually react to were the ones you got by reloading history. The feedback signal we most wanted was, in practice, almost unreachable. Our production data says as much: reactions across the entire history of the product are close to none, which is a very polite way of saying the button was decorative.

The fix is a trailing event. After the completion event, we persist the message and send its id:

saved_message_id = await self._save_assistant_message_async(accumulated_message)
# Sent after "completion" on purpose: the row only exists once the stream
# is done, and clients must not wait on this event to finish rendering.
yield json.dumps({"type": "message_saved", "message_id": saved_message_id})

The ordering comment is the important part. It would be tidier to send an id first and have one authoritative event that means "the reply is complete and here is everything about it". It would also mean that a failure to write a row could block a perfectly good answer from being displayed. Rendering must never depend on persistence. So the reply completes, and the id catches up a moment later to enable a secondary affordance.

Retry is only safe before the first tool runs

Streams break. Upstream connections drop, providers return transient server errors, networks do network things. The obvious response is to retry the turn.

The obvious response is dangerous. Our assistant creates invoices, transactions and bill payments. Retrying a turn that already executed create_transaction does not produce a better answer, it produces two transactions.

So the retry rule is guarded by two pieces of state that the streaming loop tracks anyway:

is_transient = isinstance(e, APIError) and getattr(e, "type", None) == "server_error"
if is_transient and attempt < MAX_STREAM_RETRIES and not tool_executed:
    await asyncio.sleep(attempt + 1)
    continue

tool_executed flips to true the moment any tool call returns output. After that, the turn is no longer a pure function of its input, and we do not get to pretend otherwise. The same rule applies to a dropped connection, with one addition: we also refuse to retry if any text has already been streamed, because the user has seen it and restarting the answer from the top is its own kind of broken.

This is the same instinct that runs through the rest of our system. When something has already touched the outside world, the only safe move is forward.

A partial answer beats a red error

If we cannot retry, we salvage. When a stream drops after real content has arrived, we do not throw the text away. We save what we have, emit a completion event carrying it, and follow with the message id, exactly as a successful turn would. The user sees a short reply instead of an error, and it stays in their history like any other message.

Only a drop with nothing accumulated becomes a visible error, and even then it is a plain sentence asking them to try again, not a stack trace.

The principle: a turn that produced something should end as a message, and a turn that produced nothing should end as an apology. There is no third case where the user has to guess.

The connection you forgot about

My favourite bug in this system had nothing to do with the model.

Persisting messages from an async loop means the ORM runs inside executor threads. Django recycles its database connections on request start and request end signals, and those signals never fire for threads you spawned yourself. So a long turn can leave a pooled connection idle for the entire duration of the model call, long enough for the server on the other end to close it. The next query then fails with the connection already being gone, on a request that was otherwise perfectly healthy.

The fix is small and specific:

def _retry_on_stale_connection(func):
    """Recover from a Postgres connection dropped while idle.

    Each wrapped helper performs a single create or save with no committed
    side effects before the failure, so dropping the dead connection and
    retrying exactly once is idempotent.
    """
    @wraps(func)
    def wrapper(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except (OperationalError, InterfaceError):
            close_old_connections()
            return func(*args, **kwargs)
    return wrapper

Note the justification in the docstring, not just the mechanism. The retry is allowed because the wrapped operation is a single write with nothing committed before it. That is the difference between a safe retry and a hopeful one, and it is worth writing down where the next person will read it.

Every timeout in the stack has to outlive the slowest turn

Go back to the latency distribution: p99 around thirty seconds. Now consider a worker request timeout of twenty five seconds, which is a perfectly reasonable default for a transactional API.

That combination is a machine for producing mysterious failures. The worker is killed mid-turn, so the async loop dies without cleanup, orphaned tool subprocesses linger, and the user sees a generic error on the exact requests that were doing the most work. Nothing in the application logs says "timeout", because the process that would have logged it is gone.

The lesson generalises into a rule we now apply deliberately:

1. The request timeout must exceed the slowest turn you are willing to serve, not the median one.
2. Every inner timeout, tool calls especially, must be shorter than the request timeout, so a slow tool fails as a catchable exception rather than as a killed process.
3. Streaming endpoints do not belong in the same worker budget as transactional ones. A request that holds a worker for thirty seconds and a request that returns in eighty milliseconds should not compete for the same slots.

There is a fourth rule we are rolling out now, and it is the least obvious: send a byte early. Proxies in front of an API commonly retry a request that has not yet produced any response bytes. For a normal endpoint that is a helpful safety net. For a streaming endpoint that may spend twenty seconds thinking before its first token, it is a duplicate-execution hazard. Flushing a trivial first chunk before the agent starts closes that window.

The model is the easy part

None of this is about prompting. Every hard decision here is an old distributed systems decision wearing new clothes: what is idempotent, what may be retried, what may be reordered, what must never block rendering, and which timeout is the shortest one in the chain.

If you are building this, the shortest useful summary I can give is: write down your event contract before you write the streaming loop, decide for each event whether it is required or additive, and decide for each failure whether it produces a message or an apology. The token streaming will work on the first try. Everything else is the actual product.