Skip to main content
RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared

RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared

KU
· Founder & Chief Editor
• 7 min read•Updated •30 views

Our support bot kept citing half a sentence and missing the answer that sat two lines below. The culprit wasn't the model, it was how we split the docs.

Key takeaways

  • Use recursive chunking for structured docs such as markdown and HTML, at roughly 500 to 1,000 tokens with about 10 to 15 percent overlap.
  • Keep fixed-size splitting as the baseline you measure against.
  • Reserve semantic chunking for unstructured text like transcripts, where it can earn its extra embedding cost.

For most RAG systems, use recursive chunking: split on headings, then paragraphs, then sentences, at roughly 500 to 1,000 tokens with about 10 to 15% overlap. Keep fixed-size splitting as the baseline you measure against, and use semantic chunking only for unstructured text such as transcripts, where it justifies embedding every sentence.

Verified against

langchain-text-splitters 1.1.3 (released 2026-10-02) and OpenAI text-embedding-3-large, against the LangChain text splitter docs and the OpenAI embeddings guide. Checked October 2026.

The complaint that started this came from support: the bot answered "how do I rotate an API key" with the first half of the right paragraph and then trailed off, missing the actual POST /keys/rotate example that lived in the very next sentence. Retrieval was finding the right document. The chunk just got cut in the wrong place. We had been splitting on a naive 512-token window, and the answer straddled a boundary. The fix was not a better model. It was a better split.

The three strategies at a glance

StrategySplits onIngest costTuning effortBest forMain failure mode
Fixed-sizeEvery N tokensLowestSize and overlap onlyBaselines, uniform textCuts mid-sentence, mid-table, mid-code
RecursiveHeadings, paragraphs, lines, wordsLowSeparator list plus sizeMarkdown, HTML, docs sitesOversized sections fall back to crude cuts
SemanticEmbedding distance between sentencesHigh (embeds every sentence)Corpus-specific thresholdTranscripts, PDFs without structureThreshold too low or too high

Fixed-size chunking, and why it's the baseline

Fixed-size splitting cuts every N tokens with some overlap. It is dumb and it is fast, and it is the right place to start because you can measure everything else against it.

chunking/fixed.py
def fixed_chunks(text, size=512, overlap=64, enc=None):
    toks = enc.encode(text)
    step = size - overlap
    return [enc.decode(toks[i:i + size])
            for i in range(0, len(toks), step)]

The overlap matters more than people think. With overlap=0, an answer that lands on a boundary gets bisected, which was exactly our bug. An overlap of 64 tokens (12.5% of a 512 window) catches most straddling answers, but every overlapping token is stored and embedded twice, so the index grows by roughly the overlap ratio and you get near-duplicate hits to dedupe at query time. Fixed chunking also ignores structure entirely: it will happily split mid-code-block or mid-table, which for technical docs is exactly where it hurts.

Recursive chunking respects structure

Recursive character splitting (LangChain popularized it, but it is a 30-line idea) tries a priority list of separators and only falls back to a cruder one when a chunk is still too big. LangChain's default list is ["\n\n", "\n", " ", ""]; for markdown we prepend heading markers so sections split before paragraphs. Headings and paragraphs stay intact, and code fences usually survive.

One trap catches almost everyone: RecursiveCharacterTextSplitter(chunk_size=800) counts characters, because its default length_function is len. If you mean tokens, build it from a tokenizer:

chunking/recursive.py
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=800,      # tokens, not characters
    chunk_overlap=100,
    separators=["\n## ", "\n### ", "\n\n", "\n", " ", ""],
)
chunks = splitter.split_text(doc)

On our heavily structured docs, moving from fixed to recursive was the single biggest retrieval improvement we made, with no other change. The reason is simple: respecting headings kept each chunk about one topic instead of a blend of two. If your corpus is well-formatted markdown or HTML, recursive is probably where you should stop.

Semantic chunking, and when it earns the cost

Semantic chunking embeds each sentence and starts a new chunk when the cosine distance between consecutive sentences spikes past a threshold, the idea being that a topic shift shows up as an embedding shift. It is appealing for messy prose (transcripts, PDFs with no clean headings) where structural separators do not exist.

chunking/semantic.py
import numpy as np

def semantic_chunks(sentences, embed, threshold=0.35):
    embs = embed(sentences)  # OpenAI embeddings are unit length, so dot = cosine
    chunks, cur = [], [sentences[0]]
    for i in range(1, len(sentences)):
        dist = 1 - np.dot(embs[i], embs[i - 1])
        if dist > threshold:
            chunks.append(" ".join(cur)); cur = []
        cur.append(sentences[i])
    chunks.append(" ".join(cur))
    return chunks

The catch is cost and tuning. You embed every sentence at index time, then embed the resulting chunks again, so ingest spend is a multiple of recursive's. The threshold is corpus-specific: set it too low and you get one-sentence chunks, too high and you are back to giant blobs. On our structured docs, semantic chunking did not measurably beat recursive, so it was not worth the ingest cost. On raw, unformatted call transcripts it clearly did better, because there were no headings for recursive splitting to use.

The variable that beat all three: chunk size

Across every strategy, chunk size moved our results more than strategy choice did. Very small chunks (around 256 tokens) lost context and answers felt clipped; very large ones (over 1,000 tokens) retrieved too much noise and the generator's answers got vaguer. For our text-embedding-3-large setup the sweet spot sat in the 700 to 900 token range with about 100 tokens of overlap, comfortably under the model's 8,192-token input limit. Your corpus will land somewhere else, so sweep the size before you obsess over the algorithm. When retrieval still looks wrong after that, work through why RAG retrieval returns garbage before blaming the splitter.

Questions people ask

What is the best chunk size for RAG?

There is no universal answer, but 500 to 1,000 tokens with 10 to 15% overlap is a sensible starting range for documentation. Smaller chunks lose surrounding context; larger ones dilute the embedding with unrelated text and feed the generator noise. Sweep three or four sizes against a fixed question set with known answer passages and pick the one with the best recall, because the right number depends on your corpus and embedding model.

Is semantic chunking better than recursive chunking?

Only on text without usable structure. On markdown, HTML and docs sites, recursive splitting on headings and paragraphs already keeps chunks on one topic, so semantic chunking adds embedding cost for little or no retrieval gain. On transcripts, chat logs and PDFs with no headings, semantic boundaries can beat recursive splitting, because there are no structural separators for it to use.

Does LangChain chunk_size count tokens or characters?

Characters, by default. RecursiveCharacterTextSplitter measures length with its length_function, which defaults to Python's len. To size chunks in tokens, construct it with RecursiveCharacterTextSplitter.from_tiktoken_encoder(...) and pass encoding_name or model_name. Mixing the two up is common: an 800-character chunk holds only a few hundred tokens of English, far smaller than intended.

The call we'd make

For clean, structured docs, use recursive splitting at about 800 tokens with 100 overlap and do not overthink it. It gets you most of the way at a fraction of semantic's ingest cost. Reserve semantic chunking for genuinely unstructured text where separators fail. And before you A/B any of this, build a 50-question eval set with known-good source passages and measure Recall@5 with an offline retrieval evaluation harness, because otherwise you are tuning by vibes, and chunking is exactly the place where vibes lie to you.

Sources

Explore topics:LLMAI
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2026). RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/rag-chunking-strategies-fixed-semantic-recursive

Markdown

[RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared](https://www.devopsness.com/blog/rag-chunking-strategies-fixed-semantic-recursive) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)

KU

Kiril Urbonas

Founder & Chief Editor

605 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.