12.8.5 — Streaming to the User

In its first version, "Ask Cartwise" waited for the model's complete answer and then showed it. For a typical question with one tool call, that meant a blank chat bubble with a spinner for six or seven seconds. The analytics were brutal: about a fifth of customers closed the chat before the answer appeared, and most of them then phoned the support line, which costs Cartwise far more than an AI answer.

The model was not slow at producing words; 12.7.6 showed it produces tens of tokens a second. The problem was that the interface hid all of that work until the very end. The fix was to show the answer as it is generated, starting within about a second, and to show what the assistant is doing ("Looking up order 4471…") while it waits on tools. Total time did not change at all. Abandonment fell sharply, because people tolerate waiting while they can see progress far better than they tolerate a blank screen.

That is streaming, and it is how every chat assistant and AI coding tool you have used works. This page builds it from the model API to the browser, then explains how those interfaces turn a stream of tokens into formatted text, live tool activity and interactive components.

1. Streaming from the model

Instead of one response at the end, the API sends a sequence of small events as generation proceeds: "a message has started", "here are the next few characters of text", "a tool call has started", "here is part of its arguments", "the message is finished, and here are the token counts". The Python SDK wraps this in a helper:

import anthropic
client = anthropic.Anthropic()

with client.messages.stream(                                   # (1)
    model="claude-opus-5-5",
    max_tokens=2048,
    system=SUPPORT_SYSTEM_PROMPT,
    messages=messages,
) as stream:
    for text in stream.text_stream:                            # (2) arrives a few tokens at a time
        print(text, end="", flush=True)
    final = stream.get_final_message()                         # (3) the complete response
print(final.stop_reason, final.usage.output_tokens)

Line (1) opens a streaming request. Line (2) yields each new piece of text as soon as it arrives, typically every few tens of milliseconds. Line (3) returns the complete message once the stream ends, assembled for you, with the same content, stop_reason and usage as a normal call, so the rest of your code (logging, stop-reason handling, tool loops) does not change.

Two numbers from 12.7.6 now describe the experience. Time to first token is the wait before anything appears, mostly the prefill of the prompt. Tokens per second is how fast text then flows. A user's sense of speed depends far more on the first than the second, which is why streaming transforms how fast an assistant feels without making it any faster.

2. From your server to the browser

The browser should never call the model API directly, because that would expose your API key, as 12.8.1 explained. So the stream travels in two hops: the model provider streams to Cartwise's backend, and the backend streams to the customer's browser.

Streaming in two hops. The backend receives the model's stream, runs any tools, and relays a simple stream of its own design to the browser over server-sent events, including status events of its own while tools run.

The standard way to send a stream to a browser is server-sent events, SSE, from Chapter 5.8: an ordinary HTTP response that stays open, into which the server writes small text messages, each a line starting data: followed by a blank line. It works through normal proxies and load balancers, and browsers support it natively. WebSockets, which allow messages in both directions, are used when the client also needs to send messages mid-stream, but for "server sends the answer as it grows", SSE is the simpler, usual choice.

Here is the backend relay, using FastAPI, a widely used Python web framework, and the tool loop from 12.8.4:

import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

def sse(event: str, data: dict) -> str:
    return f"event: {event}\ndata: {json.dumps(data)}\n\n"            # (1)

@app.post("/chat")
def chat(req: ChatRequest):
    def events():
        messages = build_messages(req)
        for _ in range(6):
            with client.messages.stream(model="claude-opus-5-5", max_tokens=2048,
                                        system=SUPPORT_SYSTEM_PROMPT, tools=TOOLS,
                                        messages=messages) as stream:
                for text in stream.text_stream:
                    yield sse("text", {"delta": text})                  # (2)
                final = stream.get_final_message()
            if final.stop_reason != "tool_use":
                yield sse("done", {"stop_reason": final.stop_reason})
                return
            messages.append({"role": "assistant", "content": final.content})
            results = []
            for block in (b for b in final.content if b.type == "tool_use"):
                yield sse("status", {"text": describe(block)})           # (3) "Looking up order 4471…"
                results.append(run_tool_safely(block))
            messages.append({"role": "user", "content": results})
        yield sse("done", {"stop_reason": "too_many_steps"})
    return StreamingResponse(events(), media_type="text/event-stream")   # (4)

Line (1) formats one SSE message: an event name, a line of JSON data, and the blank line that ends it. Line (2) relays each piece of the model's text the moment it arrives. Line (3) is the part that makes waits feel short: before running each tool, the backend sends its own event describing what is happening, which the interface shows in place of a silent spinner. describe turns get_order_status("4471") into plain words. Line (4) returns the generator as a streaming HTTP response with the SSE content type.

In the browser, the interface reads the stream and appends each piece of text to the chat bubble. Written in TypeScript, since that is what browser code is:

const res = await fetch("/chat", { method: "POST", body: JSON.stringify(req), signal });  // (1)
const reader = res.body!.pipeThrough(new TextDecoderStream()).getReader();
let buffer = "";
while (true) {
  const { value, done } = await reader.read();                     // (2)
  if (done) break;
  buffer += value;
  const events = buffer.split("\n\n");                              // (3)
  buffer = events.pop()!;                                           //     keep any partial event
  for (const raw of events) {
    const event = raw.match(/^event: (.*)$/m)?.[1];
    const data = JSON.parse(raw.match(/^data: (.*)$/m)?.[1] ?? "{}");
    if (event === "text") bubble.append(data.delta);                // (4)
    if (event === "status") showStatus(data.text);
    if (event === "done") finish(data.stop_reason);
  }
}

Line (1) sends the request with an abort signal, from Chapter 6.3.1, so that a "Stop" button can cancel generation; the backend should notice the closed connection and stop the model stream too, so you stop paying for tokens nobody will read. Line (2) reads chunks as they arrive. Line (3) splits on the blank line that ends each SSE message, keeping any incomplete message in the buffer for the next chunk, because network chunks do not line up with message boundaries. Line (4) appends text to the bubble or updates the status line.

3. Rendering text that is not finished yet

Streamed answers are usually Markdown: headings, bullet lists, bold text, code blocks. Rendering Markdown that is still arriving is trickier than it sounds, because at any moment the text may be half of something. A code block has opened with three backticks but not closed, so everything after it would render as code; a table has its header row but not its body; a link is [track your parcel](https://car. Rendering naively makes the screen flicker as the formatting jumps.

The usual approach is to render optimistically: on each update, re-render the text so far, temporarily closing anything left open (add the missing closing backticks for display, hold back a half-written link until it is complete), and smooth updates by rendering at most every 30 to 50 milliseconds instead of on every token. Chat interfaces that look polished are doing this carefully; it is ordinary front-end work of the kind in Part 6, not anything special to AI.

Streaming structured output

When the model returns structured output from 12.8.3, for example a refund form to pre-fill, you may still want to show fields as they arrive rather than waiting for the closing brace. That requires partial JSON parsing: reading JSON that is cut off mid-way, such as {"order_id": "4471", "reason": "dama, as "order_id is complete, reason is in progress". Libraries for this exist in most languages, and the SDKs expose partial tool arguments as they stream. One caution from 12.8.3 applies doubly here: a partial value is not a validated value, so never act on structured output until it is complete and has passed validation.

4. Interfaces that render components, not just text

Many AI products do something that looks like magic: in the middle of a conversation, a code editor panel opens beside the chat with the generated code in it, or a document appears in a side pane, or a booking widget, a chart, or a form appears instead of a paragraph. A natural question is whether the model is somehow generating that interface on the fly. It is not. Here is what actually happens.

How AI interfaces "open" editors, documents and widgets. Engineers build the components in advance; the model only chooses one, through a tool call or a tagged block, and supplies its contents; the client does the rendering.

The components are built in advance by product engineers, as ordinary interface code: a code editor (often an embedded editor component such as Monaco, the editor behind VS Code), a document pane, a form, a card. Each one is described to the model as a tool or a tagged output with a schema, exactly like the tools of 12.8.4: "create_code_file(language, filename, content): use when the user asks for code longer than a few lines". During the conversation, the model decides to emit that signal and streams the arguments. The client sees the signal, opens the matching component, and streams the arguments into it, which is why you can watch code appear line by line inside an editor. The model is choosing from a fixed menu of components and filling in their contents, not inventing interface code, and that is what makes the result reliable enough to ship: every component has been designed, tested and secured beforehand.

For Cartwise, the same pattern means "Ask Cartwise" can answer "where is my order?" with an order-tracking card showing the courier's map, by calling a show_order_card tool, rather than a paragraph describing it. The card is rendered from real data the backend looked up, not from anything the model wrote, which also means the model cannot misstate the delivery date inside it.

What AI coding tools add

AI code editors use the same building blocks, with a few extra mechanisms worth knowing:

• Inline suggestions ("ghost text") come from a separate, small, fast model that is triggered as you type, after a short pause (a debounce, often a few hundred milliseconds), so that it is not called on every keystroke. It is usually given the code before and after the cursor and trained to fill the gap, called fill-in-the-middle. Speed matters more than depth here, so this model is far smaller than the one in the chat panel.
• Edits arrive as structured changes, not whole files: the model emits blocks like "replace these lines with these lines" through a tool, and the editor shows them as a diff you can accept or reject. Applying a precise edit is cheaper and safer than rewriting a file.
• Agent modes stream their steps: each tool call (read a file, run tests, search the codebase) appears in the interface as it starts and finishes, with its result. Showing the sequence is what makes a multi-minute agent run understandable instead of a long silence, the same principle as the status events above. Chapter 12.15.5 builds a code assistant end to end.

5. When streams go wrong

Streams fail in ways one-shot requests do not, so plan for each:

• The connection drops mid-answer. Mobile networks do this constantly. Keep what was shown, mark it as interrupted, and offer "continue" or "retry" rather than discarding it.
• The error arrives after text has started. A rate limit or server error can occur after the first sentence has been shown. The interface must be able to show an error inside a partly written answer.
• The stop reason is not end_turn. If generation stopped at max_tokens, the visible answer is truncated, and the interface should say so; a refusal mid-stream needs a graceful message.
• Moderation after display. Some systems check output for safety or policy violations. If the check runs only on the finished text, problematic text may already have been shown. Either check in small chunks as you stream, or hold back the last sentence or so until it is checked.
• Cancelled streams still cost money unless you stop them. When the user presses Stop or closes the page, cancel the upstream model request.

Recall

• Streaming shows the answer as it is generated; time to first token, not total time, decides how fast it feels. Status events while tools run ("Looking up order 4471…") make waits tolerable.
• The SDK's messages.stream yields text as it arrives and assembles the final message, so stop-reason handling and logging are unchanged.
• The browser never calls the model directly: the provider streams to your backend, which relays its own event stream over server-sent events (lines starting data:, ended by a blank line). Buffer partial events; support cancellation with an abort signal and stop the upstream request.
• Render unfinished Markdown optimistically and at a steady rate; parse partial JSON for display only, never act on it until complete and validated.
• Interfaces that open editors, documents, forms or cards use pre-built components that the model selects through tool calls or tagged output; the client renders them. Coding tools add a small fill-in-the-middle model for inline suggestions, structured diffs for edits, and streamed agent steps. Handle dropped connections, mid-stream errors, truncation and moderation.

Self-test: Why does streaming reduce abandonment even though the answer takes just as long? Why must the browser split on the blank line and keep the last piece in a buffer? When the model "opens a code editor", which parts were written by the model and which by engineers?

Next: 12.8.6 Evals — every page so far has said "measure it"; the next one is how, for a system whose answers are open-ended text and change whenever a prompt, a model or a document changes.