LLM Streaming UX: Backpressure, Cancellation, Partial Results

Stream LLM output over Server-Sent Events and wire every client disconnect to an abort of the upstream model call, or you keep paying for tokens nobody reads. Then turn off proxy buffering (X-Accel-Buffering: no), send a : keep-alive comment every 15 seconds, await drain when writes back up, and keep partial output marked incomplete.

Stream only where a human is watching

For chat and long-form generation, streaming changes how fast the product feels: the user sees activity immediately and can interrupt when it goes wrong. For batch jobs and structured extraction it adds complexity without value.

SSE is the transport, and proxies are the trap

Server-Sent Events is the boring, web-native choice: plain HTTP, one direction, debuggable with curl -N. WebSockets are bidirectional, which is overkill for tokens; save them for real-time voice. On the wire each event is a data: line followed by a blank line:

data: {"type":"token","content":"Hello"}

data: {"type":"token","content":" world"}

data: {"type":"done","total_tokens":42}

Two proxy behaviors will bite you:

Buffering. nginx's proxy_buffering is on by default, so the user sees nothing until generation finishes. The nginx docs say a X-Accel-Buffering: no response header disables it per response, so send it with Content-Type: text/event-stream and Cache-Control: no-cache.

Idle reaping. An AWS ALB closes connections idle for 60 seconds by default (configurable from 1 to 4000), and nginx's proxy_read_timeout also defaults to 60 seconds. A model that pauses during a long tool call exceeds that easily. The WHATWG spec itself suggests a comment line (starting with :) "every 15 seconds or so"; clients ignore it and the idle timer resets.

Cancellation: the pattern that pays for itself

When the user closes the tab or hits stop, the HTTP request is over, but the model call keeps running and billing unless you abort it.

There is a Node trap here that breaks most copy-pasted examples. Since Node v16, the request's 'close' event fires when the request body has been fully read, not when the socket drops. With a JSON body, req.on("close", abort) aborts the model call almost immediately. Listen on the response instead: res's 'close' fires when the response completes or the connection terminates early, so check writableEnded to tell the two apart.

import express from "express";
import OpenAI from "openai";

const app = express();
const openai = new OpenAI();
app.use(express.json());

app.post("/chat", async (req, res) => {
  const controller = new AbortController();
  res.on("close", () => {
    if (!res.writableEnded) controller.abort(); // client went away mid-stream
  });

  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    "X-Accel-Buffering": "no",
  });
  const keepAlive = setInterval(() => res.write(": keep-alive\n\n"), 15_000);

  try {
    const stream = await openai.chat.completions.create(
      { model: process.env.LLM_MODEL, messages: req.body.messages, stream: true },
      { signal: controller.signal }
    );
    for await (const chunk of stream) {
      const content = chunk.choices[0]?.delta?.content;
      if (content) await writeEvent(res, { type: "token", content });
    }
    await writeEvent(res, { type: "done" });
  } catch (err) {
    if (!controller.signal.aborted) {
      await writeEvent(res, { type: "error", error: "upstream failed", partial: true });
    }
  } finally {
    clearInterval(keepAlive);
    res.end();
  }
});

app.listen(3000);

Keep the model name in config. Hard-coded names age badly: OpenAI's deprecations page lists the gpt-4o snapshots for shutdown on January 14, 2027 (see the OpenAI shutdown calendar).

EventSource only does GET, so for a POST with a message body the browser reads the stream with fetch and cancels it with its own AbortController. Calling abort() on stop closes the connection, which fires res 'close' on the server:

const stop = new AbortController();
const res = await fetch("/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ messages }),
  signal: stop.signal,
});
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  render(value); // split on "\n\n", parse each data: line
}
// stopButton.onclick = () => stop.abort();

In Python with FastAPI or Starlette, check await request.is_disconnected() inside the generation loop and close the provider stream when it returns true.

Three timeouts, not one

Each catches a different failure; whichever fires first aborts the upstream call and emits a structured error event:

• First-token timeout, around 30 seconds. No token at all usually means provider trouble. Fail fast.
• Inter-token timeout, around 15 seconds. Catches streams that stall without erroring.
• Total timeout, a few minutes. Long enough for real long generations, short enough that hung requests do not pile up.

These are our starting values, not vendor limits; tune them to your model's observed latency.

Backpressure: the slow reader problem

A phone on bad Wi-Fi reads slowly. The model produces tokens faster than the socket drains, and anything that buffers without bound will exhaust memory across a few thousand slow connections. Node's stream docs are direct: write() returns false when the buffer is past highWaterMark, and you should not write again until 'drain'.

function writeEvent(res, event) {
  if (res.writableEnded || res.destroyed) return Promise.resolve();
  const ok = res.write(`data: ${JSON.stringify(event)}\n\n`);
  return ok ? Promise.resolve() : new Promise((r) => res.once("drain", r));
}

The await pauses the for await loop, which stops pulling chunks from the model stream, so the server runs at the client's pace. In Python, an async generator gives the same effect because yield suspends until the consumer pulls; if you add a decoupling queue, give it a maxsize so put blocks when the client falls behind, and cancel the filler task in finally.

Partial results and word-boundary buffering

When the stream fails after 200 tokens, you can discard the partial, keep it marked incomplete, or auto-retry. Discarding is bad UX: the user watched text appear, then vanish. Auto-retry is hard, because you cannot reliably prompt a model to continue from token N. Our recommendation: keep the partial and send a final {"type":"error","partial":true} event; the client shows what arrived plus an "incomplete" badge and a retry button.

Token-by-token rendering also looks janky, so buffer on the server to word boundaries, and for markdown to paragraph or list-item boundaries.

Failure modes and the exact symptoms

Where this breaks

Aborting the HTTP request stops generation and further billing on providers that tie generation to the connection, but you are normally still billed for input tokens and anything already generated; check your provider's docs before projecting savings. Serverless platforms with response buffering or hard duration caps may not stream at all, or cut long generations. Compression middleware can buffer the stream regardless of your headers, so exclude text/event-stream from it. And if you need resumable streams across reconnects, SSE's Last-Event-ID gives you the plumbing, but you must persist generated output server-side to replay it. For cutting the cost side of the same requests, pair this with prompt caching.

Questions people ask

Why does my SSE stream arrive all at once behind nginx?

nginx buffers proxied responses by default (proxy_buffering on), so tokens collect until the upstream finishes. Send the X-Accel-Buffering: no response header from your app, or set proxy_buffering off for that location, and make sure no compression layer sits in the path. The client should then receive each data: event as soon as your server writes it.

Does closing the browser tab stop LLM token billing?

Not by itself. The provider keeps generating until your server aborts the upstream request. In Node, listen for 'close' on the response object and call abort() on the AbortController passed to the SDK, checking res.writableEnded first. In Starlette or FastAPI, poll request.is_disconnected() and close the provider stream. Without this, every abandoned generation is billed to completion.

Why does my LLM stream disconnect after 60 seconds?

Something in the path closes idle connections. AWS Application Load Balancers default to a 60-second idle timeout, and nginx's proxy_read_timeout also defaults to 60 seconds. A model pausing for a tool call sends nothing for that long. Write an SSE comment line such as : keep-alive every 15 seconds, which the WHATWG spec recommends, or raise both timeouts.

The call we'd make

Use SSE, and treat cancellation as non-negotiable: every client disconnect must abort the upstream call, and in Node that means the response's 'close' event, not the request's. Add three timeouts, respect backpressure, buffer to word boundaries, and keep partial results with an explicit "incomplete" marker. None of these patterns is exotic, and all are easy to skip until the bill or the mystery disconnects show up.

Sources

• nginx proxy module: proxy_buffering and X-Accel-Buffering
• WHATWG HTML: Server-sent events
• Node.js docs: HTTP (IncomingMessage and ServerResponse close events)
• AWS: Application Load Balancer attributes (connection idle timeout)