RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared
Our support bot kept citing half a sentence and missing the answer that sat two lines below. The culprit wasn't the model, it was how we split the docs.
Key takeaways
- Use recursive chunking for structured docs such as markdown and HTML, at roughly 500 to 1,000 tokens with about 10 to 15 percent overlap.
- Keep fixed-size splitting as the baseline you measure against.
- Reserve semantic chunking for unstructured text like transcripts, where it can earn its extra embedding cost.
On this page
For most RAG systems, use recursive chunking: split on headings, then paragraphs, then sentences, at roughly 500 to 1,000 tokens with about 10 to 15% overlap. Keep fixed-size splitting as the baseline you measure against, and use semantic chunking only for unstructured text such as transcripts, where it justifies embedding every sentence.
Verified against
langchain-text-splitters 1.1.3 (released 2026-10-02) and OpenAI text-embedding-3-large, against the LangChain text splitter docs and the OpenAI embeddings guide. Checked October 2026.
The complaint that started this came from support: the bot answered "how do I rotate an API key" with the first half of the right paragraph and then trailed off, missing the actual POST /keys/rotate example that lived in the very next sentence. Retrieval was finding the right document. The chunk just got cut in the wrong place. We had been splitting on a naive 512-token window, and the answer straddled a boundary. The fix was not a better model. It was a better split.
The three strategies at a glance
| Strategy | Splits on | Ingest cost | Tuning effort | Best for | Main failure mode |
|---|---|---|---|---|---|
| Fixed-size | Every N tokens | Lowest | Size and overlap only | Baselines, uniform text | Cuts mid-sentence, mid-table, mid-code |
| Recursive | Headings, paragraphs, lines, words | Low | Separator list plus size | Markdown, HTML, docs sites | Oversized sections fall back to crude cuts |
| Semantic | Embedding distance between sentences | High (embeds every sentence) | Corpus-specific threshold | Transcripts, PDFs without structure | Threshold too low or too high |
Fixed-size chunking, and why it's the baseline
Fixed-size splitting cuts every N tokens with some overlap. It is dumb and it is fast, and it is the right place to start because you can measure everything else against it.
def fixed_chunks(text, size=512, overlap=64, enc=None):
toks = enc.encode(text)
step = size - overlap
return [enc.decode(toks[i:i + size])
for i in range(0, len(toks), step)]
The overlap matters more than people think. With overlap=0, an answer that lands on a boundary gets bisected, which was exactly our bug. An overlap of 64 tokens (12.5% of a 512 window) catches most straddling answers, but every overlapping token is stored and embedded twice, so the index grows by roughly the overlap ratio and you get near-duplicate hits to dedupe at query time. Fixed chunking also ignores structure entirely: it will happily split mid-code-block or mid-table, which for technical docs is exactly where it hurts.
Recursive chunking respects structure
Recursive character splitting (LangChain popularized it, but it is a 30-line idea) tries a priority list of separators and only falls back to a cruder one when a chunk is still too big. LangChain's default list is ["\n\n", "\n", " ", ""]; for markdown we prepend heading markers so sections split before paragraphs. Headings and paragraphs stay intact, and code fences usually survive.
One trap catches almost everyone: RecursiveCharacterTextSplitter(chunk_size=800) counts characters, because its default length_function is len. If you mean tokens, build it from a tokenizer:
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
encoding_name="cl100k_base",
chunk_size=800, # tokens, not characters
chunk_overlap=100,
separators=["\n## ", "\n### ", "\n\n", "\n", " ", ""],
)
chunks = splitter.split_text(doc)
On our heavily structured docs, moving from fixed to recursive was the single biggest retrieval improvement we made, with no other change. The reason is simple: respecting headings kept each chunk about one topic instead of a blend of two. If your corpus is well-formatted markdown or HTML, recursive is probably where you should stop.
Semantic chunking, and when it earns the cost
Semantic chunking embeds each sentence and starts a new chunk when the cosine distance between consecutive sentences spikes past a threshold, the idea being that a topic shift shows up as an embedding shift. It is appealing for messy prose (transcripts, PDFs with no clean headings) where structural separators do not exist.
import numpy as np
def semantic_chunks(sentences, embed, threshold=0.35):
embs = embed(sentences) # OpenAI embeddings are unit length, so dot = cosine
chunks, cur = [], [sentences[0]]
for i in range(1, len(sentences)):
dist = 1 - np.dot(embs[i], embs[i - 1])
if dist > threshold:
chunks.append(" ".join(cur)); cur = []
cur.append(sentences[i])
chunks.append(" ".join(cur))
return chunks
The catch is cost and tuning. You embed every sentence at index time, then embed the resulting chunks again, so ingest spend is a multiple of recursive's. The threshold is corpus-specific: set it too low and you get one-sentence chunks, too high and you are back to giant blobs. On our structured docs, semantic chunking did not measurably beat recursive, so it was not worth the ingest cost. On raw, unformatted call transcripts it clearly did better, because there were no headings for recursive splitting to use.
The variable that beat all three: chunk size
Across every strategy, chunk size moved our results more than strategy choice did. Very small chunks (around 256 tokens) lost context and answers felt clipped; very large ones (over 1,000 tokens) retrieved too much noise and the generator's answers got vaguer. For our text-embedding-3-large setup the sweet spot sat in the 700 to 900 token range with about 100 tokens of overlap, comfortably under the model's 8,192-token input limit. Your corpus will land somewhere else, so sweep the size before you obsess over the algorithm. When retrieval still looks wrong after that, work through why RAG retrieval returns garbage before blaming the splitter.
Questions people ask
What is the best chunk size for RAG?
There is no universal answer, but 500 to 1,000 tokens with 10 to 15% overlap is a sensible starting range for documentation. Smaller chunks lose surrounding context; larger ones dilute the embedding with unrelated text and feed the generator noise. Sweep three or four sizes against a fixed question set with known answer passages and pick the one with the best recall, because the right number depends on your corpus and embedding model.
Is semantic chunking better than recursive chunking?
Only on text without usable structure. On markdown, HTML and docs sites, recursive splitting on headings and paragraphs already keeps chunks on one topic, so semantic chunking adds embedding cost for little or no retrieval gain. On transcripts, chat logs and PDFs with no headings, semantic boundaries can beat recursive splitting, because there are no structural separators for it to use.
Does LangChain chunk_size count tokens or characters?
Characters, by default. RecursiveCharacterTextSplitter measures length with its length_function, which defaults to Python's len. To size chunks in tokens, construct it with RecursiveCharacterTextSplitter.from_tiktoken_encoder(...) and pass encoding_name or model_name. Mixing the two up is common: an 800-character chunk holds only a few hundred tokens of English, far smaller than intended.
The call we'd make
For clean, structured docs, use recursive splitting at about 800 tokens with 100 overlap and do not overthink it. It gets you most of the way at a fraction of semantic's ingest cost. Reserve semantic chunking for genuinely unstructured text where separators fail. And before you A/B any of this, build a 50-question eval set with known-good source passages and measure Recall@5 with an offline retrieval evaluation harness, because otherwise you are tuning by vibes, and chunking is exactly the place where vibes lie to you.
Sources
Cite this article
Kiril Urbonas (2026). RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/rag-chunking-strategies-fixed-semantic-recursive
[RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared](https://www.devopsness.com/blog/rag-chunking-strategies-fixed-semantic-recursive) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)
More from AI
Explore more articles in this category
Building RAG Applications: A Complete Guide to Retrieval Augmented Generation
We've shipped four production RAG applications. Each one taught us something. The end-to-end pattern that works.
Multi-Agent AI Systems: Building Collaborative AI Applications
Multi-agent systems are mostly hype. The patterns we've seen actually deliver value, plus the ones we'd avoid until the tooling is more mature.
MLOps: Model Registry vs MLflow Tracking, And When You Need Both
Tracking experiments and shipping models are different problems. The MLOps tooling assumes one solution; production splits them. The patterns we use.
You might have missed
Evergreen posts worth revisiting.