Our support bot kept citing half a sentence and missing the answer that sat two lines below. The culprit wasn't the model, it was how we split the docs.
RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared
The complaint came from support: the bot answered "how do I rotate an API key" with the first half of the right paragraph and then trailed off, missing the actual POST /keys/rotate example that lived in the very next sentence. The retrieval was finding the right document. The chunk just got cut in the wrong place. We'd been splitting on a naive 512-token window, and the answer straddled a boundary. That sent me down a week of testing chunking strategies against our own eval set.
Fixed-size splitting cuts every N tokens with some overlap. It's dumb and it's fast, and it's the right place to start because you can measure everything else against it.
def fixed_chunks(text, size=512, overlap=64, enc=None):
toks = enc.encode(text)
step = size - overlap
return [enc.decode(toks[i:i + size])
for i in range(0, len(toks), step)]
The overlap matters more than people think. With overlap=0, an answer that lands on a boundary gets bisected, exactly our bug. Bumping overlap to 64 tokens (12.5% of a 512 window) fixed most straddling cases but inflated our index by roughly 14% and added duplicate hits we had to dedupe at query time. Fixed chunking ignores structure entirely: it'll happily split mid-code-block or mid-table, which for technical docs is where it hurts.
Recursive character splitting (LangChain popularized it, but it's a 30-line idea) tries a priority list of separators and only falls back to a cruder one when a chunk is still too big. For markdown the list is something like ["\n## ", "\n### ", "\n\n", "\n", " "]. It splits on the biggest semantic break first, so headings and paragraphs stay intact and code fences usually survive.
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=100,
separators=["\n## ", "\n### ", "\n\n", "\n", " ", ""],
)
chunks = splitter.split_text(doc)
On our docs this was the single biggest quality jump. Recall@5 on the eval set went from 0.71 (fixed) to 0.84 (recursive) with no other change. The reason is simple: our docs are heavily structured, and respecting headings kept each chunk about one topic instead of a blend of two. If your corpus is well-formatted markdown or HTML, recursive is probably where you should stop.
Semantic chunking embeds each sentence and starts a new chunk when the cosine distance between consecutive sentences spikes past a threshold, the idea being that a topic shift shows up as an embedding shift. It's appealing for messy prose (transcripts, PDFs with no clean headings) where structural separators don't exist.
import numpy as np
def semantic_chunks(sentences, embed, threshold=0.35):
embs = embed(sentences)
chunks, cur = [], [sentences[0]]
for i in range(1, len(sentences)):
dist = 1 - np.dot(embs[i], embs[i-1])
if dist > threshold:
chunks.append(" ".join(cur)); cur = []
cur.append(sentences[i])
chunks.append(" ".join(cur))
return chunks
The catch is cost and tuning. You embed every sentence at index time (roughly 4x the embedding spend versus recursive on our corpus), and the threshold is corpus-specific. Set it too low and you get one-sentence chunks; too high and you're back to giant blobs. On our structured docs semantic chunking scored 0.85 Recall@5, statistically tied with recursive, at four times the ingest cost. It wasn't worth it. On a separate corpus of raw call-center transcripts with no formatting, semantic beat recursive 0.79 to 0.68, and there it clearly earned its place.
Something worth saying loudly. Across every strategy, chunk size moved the numbers more than strategy choice did. At 256 tokens we lost context and answers felt clipped; at 1200 we retrieved too much noise and the generator's answers got vaguer. The sweet spot for our text-embedding-3-large setup was 700 to 900 tokens with about 100 tokens overlap. Test your own, but don't obsess over the algorithm before you've swept the size.
For clean, structured docs, use recursive splitting at 800 tokens with 100 overlap and don't overthink it. It gets you 90% of the way at a fraction of semantic's ingest cost. Reserve semantic chunking for genuinely unstructured text where separators fail. And before you A/B any of this, build a 50-question eval set with known-good source passages and measure Recall@5, because otherwise you're tuning by vibes, and chunking is exactly the place where vibes lie to you.
