RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared

RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared#

The complaint came from support: the bot answered "how do I rotate an API key" with the first half of the right paragraph and then trailed off, missing the actual POST /keys/rotate example that lived in the very next sentence. The retrieval was finding the right document. The chunk just got cut in the wrong place. We'd been splitting on a naive 512-token window, and the answer straddled a boundary. That sent me down a week of testing chunking strategies against our own eval set.

Fixed-size chunking, and why it's the baseline#

Fixed-size splitting cuts every N tokens with some overlap. It's dumb and it's fast, and it's the right place to start because you can measure everything else against it.

def fixed_chunks(text, size=512, overlap=64, enc=None):
    toks = enc.encode(text)
    step = size - overlap
    return [enc.decode(toks[i:i + size])
            for i in range(0, len(toks), step)]

The overlap matters more than people think. With overlap=0, an answer that lands on a boundary gets bisected, exactly our bug. Bumping overlap to 64 tokens (12.5% of a 512 window) fixed most straddling cases but inflated our index by roughly 14% and added duplicate hits we had to dedupe at query time. Fixed chunking ignores structure entirely: it'll happily split mid-code-block or mid-table, which for technical docs is where it hurts.

Recursive chunking respects structure#

Recursive character splitting (LangChain popularized it, but it's a 30-line idea) tries a priority list of separators and only falls back to a cruder one when a chunk is still too big. For markdown the list is something like ["\n## ", "\n### ", "\n\n", "\n", " "]. It splits on the biggest semantic break first, so headings and paragraphs stay intact and code fences usually survive.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100,
    separators=["\n## ", "\n### ", "\n\n", "\n", " ", ""],
)
chunks = splitter.split_text(doc)

On our docs this was the single biggest quality jump. Recall@5 on the eval set went from 0.71 (fixed) to 0.84 (recursive) with no other change. The reason is simple: our docs are heavily structured, and respecting headings kept each chunk about one topic instead of a blend of two. If your corpus is well-formatted markdown or HTML, recursive is probably where you should stop.

Semantic chunking, and when it earns the cost#

Semantic chunking embeds each sentence and starts a new chunk when the cosine distance between consecutive sentences spikes past a threshold, the idea being that a topic shift shows up as an embedding shift. It's appealing for messy prose (transcripts, PDFs with no clean headings) where structural separators don't exist.

import numpy as np

def semantic_chunks(sentences, embed, threshold=0.35):
    embs = embed(sentences)
    chunks, cur = [], [sentences[0]]
    for i in range(1, len(sentences)):
        dist = 1 - np.dot(embs[i], embs[i-1])
        if dist > threshold:
            chunks.append(" ".join(cur)); cur = []
        cur.append(sentences[i])
    chunks.append(" ".join(cur))
    return chunks

The catch is cost and tuning. You embed every sentence at index time (roughly 4x the embedding spend versus recursive on our corpus), and the threshold is corpus-specific. Set it too low and you get one-sentence chunks; too high and you're back to giant blobs. On our structured docs semantic chunking scored 0.85 Recall@5, statistically tied with recursive, at four times the ingest cost. It wasn't worth it. On a separate corpus of raw call-center transcripts with no formatting, semantic beat recursive 0.79 to 0.68, and there it clearly earned its place.

The variable that beat all three: chunk size#

Something worth saying loudly. Across every strategy, chunk size moved the numbers more than strategy choice did. At 256 tokens we lost context and answers felt clipped; at 1200 we retrieved too much noise and the generator's answers got vaguer. The sweet spot for our text-embedding-3-large setup was 700 to 900 tokens with about 100 tokens overlap. Test your own, but don't obsess over the algorithm before you've swept the size.

The call we'd make#

For clean, structured docs, use recursive splitting at 800 tokens with 100 overlap and don't overthink it. It gets you 90% of the way at a fraction of semantic's ingest cost. Reserve semantic chunking for genuinely unstructured text where separators fail. And before you A/B any of this, build a 50-question eval set with known-good source passages and measure Recall@5, because otherwise you're tuning by vibes, and chunking is exactly the place where vibes lie to you.