Streaming LLM Responses: SSE, Backpressure, and Cancellation

Streaming LLM Responses: SSE, Backpressure, and Cancellation#

We shipped streaming, watched the token dashboard, and found something ugly. Users would start a long generation, read the first paragraph, decide it was wrong, and hit stop. The UI cleared instantly, so it looked fine. But our backend kept streaming from the model for another 30 to 40 seconds, billing every token, because closing the browser tab never told the upstream request to abort. At peak we were paying for thousands of tokens per hour that no human would ever read.

Server-Sent Events, not WebSockets#

For one-directional token streaming, SSE is the right tool and WebSockets are overkill. SSE is plain HTTP, it reconnects on its own, and it plays nicely with proxies once you set the headers correctly. The three headers that matter:

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

app = FastAPI()

@app.post("/chat")
async def chat(request: Request):
    async def event_stream():
        async for chunk in run_model(request):
            yield f"data: {chunk}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(
        event_stream(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no",  # stops nginx from buffering the stream
        },
    )

That X-Accel-Buffering: no header cost us an afternoon of confusion. Without it, nginx buffers the whole response and the user sees nothing until generation finishes, which defeats the entire point of streaming.

Cancellation is the part everyone forgets#

The fix for our token waste is propagating the client disconnect all the way to the model provider. When the SSE connection drops, the request's cancellation signal has to reach the upstream HTTP call. In FastAPI you check request.is_disconnected() and you pass an abort signal into the provider SDK.

async def run_model(request: Request):
    async with anthropic.AsyncClient() as client:
        async with client.messages.stream(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            messages=build_messages(request),
        ) as stream:
            async for text in stream.text_stream:
                if await request.is_disconnected():
                    await stream.close()  # aborts the upstream request
                    return
                yield text

The stream.close() call is what actually stops the billing. It closes the underlying HTTP connection to the provider, and the provider stops generating. Before we added the disconnect check, that async for loop happily kept pulling tokens into a void. After, a user hitting stop killed the upstream request within one chunk. Token waste on abandoned generations dropped to near zero.

Backpressure: the slow reader problem#

The subtler failure is a client that reads slowly, a phone on bad wifi. The model produces tokens faster than the socket drains. If you buffer everything in memory, a few thousand of these connections will OOM the process. The async generator gives you natural backpressure because yield suspends until the consumer pulls, but only if nothing in between buffers unboundedly.

We put a bounded queue between the model loop and the socket so a stalled reader applies pressure instead of ballooning memory:

import asyncio

async def buffered_stream(source, maxsize=32):
    queue = asyncio.Queue(maxsize=maxsize)
    async def fill():
        async for item in source:
            await queue.put(item)   # blocks when the queue is full
        await queue.put(None)
    task = asyncio.create_task(fill())
    try:
        while (item := await queue.get()) is not None:
            yield item
    finally:
        task.cancel()

When the queue hits 32 items the fill coroutine blocks on queue.put, which stops pulling from the model, which is exactly the backpressure you want. A slow client slows the whole chain instead of accumulating a giant buffer. The finally: task.cancel() matters too: if the client vanishes, you don't leak the filler task.

Heartbeats keep proxies from killing you#

Idle SSE connections get reaped by load balancers. An ALB idle timeout defaults to 60 seconds, and a model that pauses while calling a tool can easily exceed that. Send a comment line as a heartbeat every 15 seconds so the proxy sees activity:

yield ": keep-alive\n\n"

Lines starting with a colon are SSE comments, ignored by the client but enough to reset the idle timer. This fixed a batch of "stream just dies at 60 seconds" reports that only showed up on longer tool-using turns.

The call we'd make#

Use SSE, not WebSockets, for token streaming. The non-negotiable is cancellation: check is_disconnected() in your generation loop and call the provider stream's close method, or you'll pay for tokens nobody reads. Put a bounded queue in the middle for backpressure, set X-Accel-Buffering: no so nginx doesn't eat your stream, and send a 15-second heartbeat to survive proxy timeouts. Get those four right and streaming stops being a source of surprise bills and mystery disconnects.