# RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared

URL: https://www.devopsness.com/blog/rag-chunking-strategies-fixed-semantic-recursive
Author: Kiril Urbonas
Published: 2026-06-29
Updated: 2026-10-31
Categories: AI
Tags: RAG, LLM, Embeddings, Search, AI

Summary: Our support bot kept citing half a sentence and missing the answer that sat two lines below. The culprit wasn't the model, it was how we split the docs.

---
# RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared

For most RAG systems, use recursive chunking: split on headings, then paragraphs, then sentences, at roughly 500 to 1,000 tokens with about 10 to 15% overlap. Keep fixed-size splitting as the baseline you measure against, and use semantic chunking only for unstructured text such as transcripts, where it justifies embedding every sentence.

> [!VERIFIED]
> langchain-text-splitters 1.1.3 (released 2026-10-02) and OpenAI text-embedding-3-large, against the LangChain text splitter docs and the OpenAI embeddings guide. Checked October 2026.

The complaint that started this came from support: the bot answered "how do I rotate an API key" with the first half of the right paragraph and then trailed off, missing the actual `POST /keys/rotate` example that lived in the very next sentence. Retrieval was finding the right document. The chunk just got cut in the wrong place. We had been splitting on a naive 512-token window, and the answer straddled a boundary. The fix was not a better model. It was a better split.

## The three strategies at a glance

| Strategy | Splits on | Ingest cost | Tuning effort | Best for | Main failure mode |
|---|---|---|---|---|---|
| Fixed-size | Every N tokens | Lowest | Size and overlap only | Baselines, uniform text | Cuts mid-sentence, mid-table, mid-code |
| Recursive | Headings, paragraphs, lines, words | Low | Separator list plus size | Markdown, HTML, docs sites | Oversized sections fall back to crude cuts |
| Semantic | Embedding distance between sentences | High (embeds every sentence) | Corpus-specific threshold | Transcripts, PDFs without structure | Threshold too low or too high |

## Fixed-size chunking, and why it's the baseline

Fixed-size splitting cuts every N tokens with some overlap. It is dumb and it is fast, and it is the right place to start because you can measure everything else against it.

```python
# chunking/fixed.py
def fixed_chunks(text, size=512, overlap=64, enc=None):
    toks = enc.encode(text)
    step = size - overlap
    return [enc.decode(toks[i:i + size])
            for i in range(0, len(toks), step)]
```

The overlap matters more than people think. With `overlap=0`, an answer that lands on a boundary gets bisected, which was exactly our bug. An overlap of 64 tokens (12.5% of a 512 window) catches most straddling answers, but every overlapping token is stored and embedded twice, so the index grows by roughly the overlap ratio and you get near-duplicate hits to dedupe at query time. Fixed chunking also ignores structure entirely: it will happily split mid-code-block or mid-table, which for technical docs is exactly where it hurts.

## Recursive chunking respects structure

Recursive character splitting (LangChain popularized it, but it is a 30-line idea) tries a priority list of separators and only falls back to a cruder one when a chunk is still too big. LangChain's default list is `["\n\n", "\n", " ", ""]`; for markdown we prepend heading markers so sections split before paragraphs. Headings and paragraphs stay intact, and code fences usually survive.

One trap catches almost everyone: `RecursiveCharacterTextSplitter(chunk_size=800)` counts **characters**, because its default `length_function` is `len`. If you mean tokens, build it from a tokenizer:

```python
# chunking/recursive.py
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=800,      # tokens, not characters
    chunk_overlap=100,
    separators=["\n## ", "\n### ", "\n\n", "\n", " ", ""],
)
chunks = splitter.split_text(doc)
```

On our heavily structured docs, moving from fixed to recursive was the single biggest retrieval improvement we made, with no other change. The reason is simple: respecting headings kept each chunk about one topic instead of a blend of two. If your corpus is well-formatted markdown or HTML, recursive is probably where you should stop.

## Semantic chunking, and when it earns the cost

Semantic chunking embeds each sentence and starts a new chunk when the cosine distance between consecutive sentences spikes past a threshold, the idea being that a topic shift shows up as an embedding shift. It is appealing for messy prose (transcripts, PDFs with no clean headings) where structural separators do not exist.

```python
# chunking/semantic.py
import numpy as np

def semantic_chunks(sentences, embed, threshold=0.35):
    embs = embed(sentences)  # OpenAI embeddings are unit length, so dot = cosine
    chunks, cur = [], [sentences[0]]
    for i in range(1, len(sentences)):
        dist = 1 - np.dot(embs[i], embs[i - 1])
        if dist > threshold:
            chunks.append(" ".join(cur)); cur = []
        cur.append(sentences[i])
    chunks.append(" ".join(cur))
    return chunks
```

The catch is cost and tuning. You embed every sentence at index time, then embed the resulting chunks again, so ingest spend is a multiple of recursive's. The `threshold` is corpus-specific: set it too low and you get one-sentence chunks, too high and you are back to giant blobs. On our structured docs, semantic chunking did not measurably beat recursive, so it was not worth the ingest cost. On raw, unformatted call transcripts it clearly did better, because there were no headings for recursive splitting to use.

## The variable that beat all three: chunk size

Across every strategy, chunk size moved our results more than strategy choice did. Very small chunks (around 256 tokens) lost context and answers felt clipped; very large ones (over 1,000 tokens) retrieved too much noise and the generator's answers got vaguer. For our `text-embedding-3-large` setup the sweet spot sat in the 700 to 900 token range with about 100 tokens of overlap, comfortably under the model's 8,192-token input limit. Your corpus will land somewhere else, so sweep the size before you obsess over the algorithm. When retrieval still looks wrong after that, work through [why RAG retrieval returns garbage](/blog/debugging-rag-retrieval-why-it-returns-garbage) before blaming the splitter.

## Questions people ask

### What is the best chunk size for RAG?

There is no universal answer, but 500 to 1,000 tokens with 10 to 15% overlap is a sensible starting range for documentation. Smaller chunks lose surrounding context; larger ones dilute the embedding with unrelated text and feed the generator noise. Sweep three or four sizes against a fixed question set with known answer passages and pick the one with the best recall, because the right number depends on your corpus and embedding model.

### Is semantic chunking better than recursive chunking?

Only on text without usable structure. On markdown, HTML and docs sites, recursive splitting on headings and paragraphs already keeps chunks on one topic, so semantic chunking adds embedding cost for little or no retrieval gain. On transcripts, chat logs and PDFs with no headings, semantic boundaries can beat recursive splitting, because there are no structural separators for it to use.

### Does LangChain chunk_size count tokens or characters?

Characters, by default. `RecursiveCharacterTextSplitter` measures length with its `length_function`, which defaults to Python's `len`. To size chunks in tokens, construct it with `RecursiveCharacterTextSplitter.from_tiktoken_encoder(...)` and pass `encoding_name` or `model_name`. Mixing the two up is common: an 800-character chunk holds only a few hundred tokens of English, far smaller than intended.

## The call we'd make

For clean, structured docs, use recursive splitting at about 800 tokens with 100 overlap and do not overthink it. It gets you most of the way at a fraction of semantic's ingest cost. Reserve semantic chunking for genuinely unstructured text where separators fail. And before you A/B any of this, build a 50-question eval set with known-good source passages and measure Recall@5 with an [offline retrieval evaluation harness](/blog/rag-retrieval-evaluation-offline-harness), because otherwise you are tuning by vibes, and chunking is exactly the place where vibes lie to you.

## Sources

- [Recursive text splitter (LangChain docs)](https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter)
- [Splitting text by tokens (LangChain docs)](https://docs.langchain.com/oss/python/integrations/splitters/split_by_token)
- [Vector embeddings guide (OpenAI API docs)](https://developers.openai.com/api/docs/guides/embeddings)
- [langchain-text-splitters release history (PyPI)](https://pypi.org/project/langchain-text-splitters/)

