# LLM Streaming UX: Backpressure, Cancellation, Partial Results

URL: https://www.devopsness.com/blog/llm-streaming-ux-backpressure-cancellation
Author: Kiril Urbonas
Published: 2026-05-07
Updated: 2026-10-31
Categories: AI
Tags: AI, LLM, Streaming, UX, Backend, Production

Summary: Streaming LLM responses is easy until the client disconnects, the model stalls, or the user cancels. The patterns that keep streaming responsive without leaking spend.

---
# LLM Streaming UX: Backpressure, Cancellation, Partial Results

Stream LLM output over Server-Sent Events and wire every client disconnect to an abort of the upstream model call, or you keep paying for tokens nobody reads. Then turn off proxy buffering (`X-Accel-Buffering: no`), send a `: keep-alive` comment every 15 seconds, await `drain` when writes back up, and keep partial output marked incomplete.

> [!VERIFIED]
> Node.js v26 `http` and `stream` docs, nginx proxy module (1.31 docs), AWS Application Load Balancer attributes, the WHATWG HTML server-sent events spec, and Starlette's request docs. Checked October 2026.

## Stream only where a human is watching

For chat and long-form generation, streaming changes how fast the product feels: the user sees activity immediately and can interrupt when it goes wrong. For batch jobs and structured extraction it adds complexity without value.

## SSE is the transport, and proxies are the trap

**Server-Sent Events** is the boring, web-native choice: plain HTTP, one direction, debuggable with `curl -N`. **WebSockets** are bidirectional, which is overkill for tokens; save them for real-time voice. On the wire each event is a `data:` line followed by a blank line:

```
data: {"type":"token","content":"Hello"}

data: {"type":"token","content":" world"}

data: {"type":"done","total_tokens":42}
```

Two proxy behaviors will bite you:

**Buffering.** nginx's `proxy_buffering` is `on` by default, so the user sees nothing until generation finishes. The nginx docs say a `X-Accel-Buffering: no` response header disables it per response, so send it with `Content-Type: text/event-stream` and `Cache-Control: no-cache`.

**Idle reaping.** An AWS ALB closes connections idle for 60 seconds by default (configurable from 1 to 4000), and nginx's `proxy_read_timeout` also defaults to 60 seconds. A model that pauses during a long tool call exceeds that easily. The WHATWG spec itself suggests a comment line (starting with `:`) "every 15 seconds or so"; clients ignore it and the idle timer resets.

## Cancellation: the pattern that pays for itself

When the user closes the tab or hits stop, the HTTP request is over, but the model call keeps running and billing unless you abort it.

There is a Node trap here that breaks most copy-pasted examples. Since Node v16, the request's `'close'` event fires when the request body has been fully read, **not** when the socket drops. With a JSON body, `req.on("close", abort)` aborts the model call almost immediately. Listen on the response instead: `res`'s `'close'` fires when the response completes or the connection terminates early, so check `writableEnded` to tell the two apart.

```javascript
// routes/chat.js
import express from "express";
import OpenAI from "openai";

const app = express();
const openai = new OpenAI();
app.use(express.json());

app.post("/chat", async (req, res) => {
  const controller = new AbortController();
  res.on("close", () => {
    if (!res.writableEnded) controller.abort(); // client went away mid-stream
  });

  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    "X-Accel-Buffering": "no",
  });
  const keepAlive = setInterval(() => res.write(": keep-alive\n\n"), 15_000);

  try {
    const stream = await openai.chat.completions.create(
      { model: process.env.LLM_MODEL, messages: req.body.messages, stream: true },
      { signal: controller.signal }
    );
    for await (const chunk of stream) {
      const content = chunk.choices[0]?.delta?.content;
      if (content) await writeEvent(res, { type: "token", content });
    }
    await writeEvent(res, { type: "done" });
  } catch (err) {
    if (!controller.signal.aborted) {
      await writeEvent(res, { type: "error", error: "upstream failed", partial: true });
    }
  } finally {
    clearInterval(keepAlive);
    res.end();
  }
});

app.listen(3000);
```

Keep the model name in config. Hard-coded names age badly: OpenAI's deprecations page lists the `gpt-4o` snapshots for shutdown on January 14, 2027 (see [the OpenAI shutdown calendar](/blog/openai-model-shutdowns-gpt-4o-january-2027)).

`EventSource` only does GET, so for a POST with a message body the browser reads the stream with `fetch` and cancels it with its own `AbortController`. Calling `abort()` on stop closes the connection, which fires `res` `'close'` on the server:

```javascript
// web/chat-client.js
const stop = new AbortController();
const res = await fetch("/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ messages }),
  signal: stop.signal,
});
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  render(value); // split on "\n\n", parse each data: line
}
// stopButton.onclick = () => stop.abort();
```

In Python with FastAPI or Starlette, check `await request.is_disconnected()` inside the generation loop and close the provider stream when it returns true.

## Three timeouts, not one

Each catches a different failure; whichever fires first aborts the upstream call and emits a structured error event:

- **First-token timeout, around 30 seconds.** No token at all usually means provider trouble. Fail fast.
- **Inter-token timeout, around 15 seconds.** Catches streams that stall without erroring.
- **Total timeout, a few minutes.** Long enough for real long generations, short enough that hung requests do not pile up.

These are our starting values, not vendor limits; tune them to your model's observed latency.

## Backpressure: the slow reader problem

A phone on bad Wi-Fi reads slowly. The model produces tokens faster than the socket drains, and anything that buffers without bound will exhaust memory across a few thousand slow connections. Node's stream docs are direct: `write()` returns `false` when the buffer is past `highWaterMark`, and you should not write again until `'drain'`.

```javascript
// routes/write-event.js
function writeEvent(res, event) {
  if (res.writableEnded || res.destroyed) return Promise.resolve();
  const ok = res.write(`data: ${JSON.stringify(event)}\n\n`);
  return ok ? Promise.resolve() : new Promise((r) => res.once("drain", r));
}
```

The `await` pauses the `for await` loop, which stops pulling chunks from the model stream, so the server runs at the client's pace. In Python, an async generator gives the same effect because `yield` suspends until the consumer pulls; if you add a decoupling queue, give it a `maxsize` so `put` blocks when the client falls behind, and cancel the filler task in `finally`.

## Partial results and word-boundary buffering

When the stream fails after 200 tokens, you can discard the partial, keep it marked incomplete, or auto-retry. Discarding is bad UX: the user watched text appear, then vanish. Auto-retry is hard, because you cannot reliably prompt a model to continue from token N. Our recommendation: keep the partial and send a final `{"type":"error","partial":true}` event; the client shows what arrived plus an "incomplete" badge and a retry button.

Token-by-token rendering also looks janky, so buffer on the server to word boundaries, and for markdown to paragraph or list-item boundaries.

## Failure modes and the exact symptoms

| Symptom | Cause | Fix |
|---|---|---|
| Whole answer arrives at once behind nginx | `proxy_buffering on` (default) | `X-Accel-Buffering: no` header or `proxy_buffering off` |
| Stream dies at exactly 60 s | ALB idle timeout or nginx `proxy_read_timeout` | `: keep-alive` comment every 15 s |
| Every request aborts right after it starts, with an SDK abort error | `req.on("close")` fired after body parsing | Listen on `res` `'close'` and check `writableEnded` |
| `ERR_STREAM_WRITE_AFTER_END` in logs | Writing after the client left | Guard writes with `res.writableEnded` and `res.destroyed` |
| Memory climbs with slow mobile clients | Ignoring `write()` returning `false` | Await `'drain'` before the next write |

## Where this breaks

Aborting the HTTP request stops generation and further billing on providers that tie generation to the connection, but you are normally still billed for input tokens and anything already generated; check your provider's docs before projecting savings. Serverless platforms with response buffering or hard duration caps may not stream at all, or cut long generations. Compression middleware can buffer the stream regardless of your headers, so exclude `text/event-stream` from it. And if you need resumable streams across reconnects, SSE's `Last-Event-ID` gives you the plumbing, but you must persist generated output server-side to replay it. For cutting the cost side of the same requests, pair this with [prompt caching](/blog/prompt-caching-production-llm-apps).

## Questions people ask

### Why does my SSE stream arrive all at once behind nginx?

nginx buffers proxied responses by default (`proxy_buffering on`), so tokens collect until the upstream finishes. Send the `X-Accel-Buffering: no` response header from your app, or set `proxy_buffering off` for that location, and make sure no compression layer sits in the path. The client should then receive each `data:` event as soon as your server writes it.

### Does closing the browser tab stop LLM token billing?

Not by itself. The provider keeps generating until your server aborts the upstream request. In Node, listen for `'close'` on the response object and call `abort()` on the `AbortController` passed to the SDK, checking `res.writableEnded` first. In Starlette or FastAPI, poll `request.is_disconnected()` and close the provider stream. Without this, every abandoned generation is billed to completion.

### Why does my LLM stream disconnect after 60 seconds?

Something in the path closes idle connections. AWS Application Load Balancers default to a 60-second idle timeout, and nginx's `proxy_read_timeout` also defaults to 60 seconds. A model pausing for a tool call sends nothing for that long. Write an SSE comment line such as `: keep-alive` every 15 seconds, which the WHATWG spec recommends, or raise both timeouts.

## The call we'd make

Use SSE, and treat cancellation as non-negotiable: every client disconnect must abort the upstream call, and in Node that means the response's `'close'` event, not the request's. Add three timeouts, respect backpressure, buffer to word boundaries, and keep partial results with an explicit "incomplete" marker. None of these patterns is exotic, and all are easy to skip until the bill or the mystery disconnects show up.

## Sources

- [nginx proxy module: proxy_buffering and X-Accel-Buffering](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering)
- [WHATWG HTML: Server-sent events](https://html.spec.whatwg.org/multipage/server-sent-events.html)
- [Node.js docs: HTTP (IncomingMessage and ServerResponse close events)](https://nodejs.org/api/http.html)
- [AWS: Application Load Balancer attributes (connection idle timeout)](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-load-balancer-attributes.html)