Semantic chunking is slower, pricier, and rarely retrieves better

LLM & Model Guides · By Caleb Sakala · June 6, 2026

Every RAG tutorial walks the same road. Start with fixed-size chunking, label it naive, then graduate to semantic chunking as the adult choice. The reasoning feels airtight: rather than cutting a document every 500 tokens, you split where the meaning shifts, so each chunk carries one coherent thought. The most careful benchmark anyone has published says that upgrade usually buys you nothing.

Renyi Qu and co-authors at Vectara ran the test the tutorials skip. Their paper, Is Semantic Chunking Worth the Computational Cost? (accepted to NAACL 2025 Findings), put three chunkers head to head: a fixed-size splitter, a breakpoint-based semantic splitter, and a clustering-based one. They scored all three on document retrieval, evidence retrieval, and answer generation. Semantic chunking cost far more to run, since every sentence gets embedded before a single cut is made, and the retrieval gains came back small and scattered, absent on several datasets entirely. A blunt fixed-size chunk of roughly 200 words kept pace with the sophisticated methods.

That result should change how you build, and most teams haven't heard it yet.

What semantic chunking does to your text

Fixed-size chunking just counts: it walks the document and cuts every N tokens, usually with a little overlap so a sentence straddling a boundary survives in both pieces. Semantic chunking measures instead, embedding each sentence, comparing neighbors, and starting a new chunk wherever the similarity between consecutive sentences drops past a threshold.

LangChain's SemanticChunker is the version most people meet first. It exposes four ways to pick that threshold: percentile, standard deviation, interquartile range, and gradient (documented here).

from langchain_experimental.text_splitter import SemanticChunker

from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(

    OpenAIEmbeddings(),

    breakpoint_threshold_type="percentile",

    breakpoint_threshold_amount=95, # cut only at the sharpest 5% of shifts

)

chunks = splitter.create_documents([raw_text])

Read that code and the hidden cost jumps out. Before you store anything, OpenAIEmbeddings() runs across every sentence in the corpus. On a 10,000-document knowledge base, that is an embedding bill you pay at ingestion, then pay again on every re-index.

Why semantic chunking rarely buys better retrieval

A few things undercut the promise. A similarity-driven splitter has no sense of how large a chunk ought to be, so a passage that changes topic every other sentence can shatter into slivers too small to retrieve usefully, while a long even-toned section swells into one oversized block. Qu's team also found the outcome swings with the embedding model and the threshold you pick, which makes the "right" configuration dataset-specific and something you have to re-tune rather than assume. The cost is not a one-time tax either. Every document added later gets embedded at the chunking stage before it is embedded again for the vector index, so you pay twice for the privilege of maybe-better boundaries.

Compare that to the baseline it is supposed to beat:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(

    chunk_size=800, # characters, tune to your embedding model

    chunk_overlap=120, # keep boundary sentences in both chunks

)

chunks = splitter.split_text(raw_text)

There is no embedding pass and no threshold to tune, and it runs in milliseconds on the same corpus the semantic splitter spends real money and minutes on. RecursiveCharacterTextSplitter tries natural separators first (paragraphs, then sentences, then words) before it resorts to a hard cut, which recovers much of the "keep coherent ideas together" benefit semantic chunking sells, at none of the price.

When semantic chunking earns its place

There is a real exception, and it matters. In domains where the document structure is dense and a wrong retrieval is costly, careful boundary detection pays off. A 2025 clinical decision-support study found that adaptive chunking aligned to logical topic boundaries reached 87% accuracy against 13% for a fixed-size baseline, a gap significant at p = 0.001. Medical records and legal contracts share the trait that makes this work: a chunk that splits a dosage from its drug, or a clause from its condition, is not slightly worse, it is dangerous. When a misplaced boundary can flip the meaning of the retrieved text, the embedding bill stops looking expensive.

So the honest rule is narrower than either camp admits. Reach for semantic or boundary-aware chunking when your documents are highly structured and retrieval mistakes carry real cost, and stay with recursive fixed-size chunking for the general case, which covers most of the support FAQs and product docs that make up RAG systems in production.

This is where the decision stops being academic. Picture a document-ingestion automation that fires every time a file lands in a shared drive, chunks it, embeds the chunks, and writes them to a vector store. Swap fixed-size for semantic chunking in that step and you have quietly doubled the embedding calls on a workflow that may run thousands of times a month. Build that pipeline in Chase Agents, where ingestion runs as a scheduled multi-step automation, and the chunking choice shows up directly in the latency and cost of every run, which turns out to be a useful forcing function: the upgrade has to prove itself before it ships.

Proving it is not hard, and skipping the proof is the actual mistake. Build a small evaluation set of real queries paired with the passage that should answer each one, chunk your corpus both ways, and compare retrieval hit rate at k=5. Twenty or thirty labeled queries is enough to see whether the expensive splitter earns its keep on your data.

def hit_rate_at_k(retriever, eval_set, k=5):

    hits = 0

    for query, gold_id in eval_set:

        results = retriever.search(query, k=k)

        if any(r.doc_id == gold_id for r in results):

            hits += 1

    return hits / len(eval_set)

# run once per chunking strategy, then compare the two numbers

If semantic chunking wins by a point or two, the embedding cost almost certainly is not worth it. If it wins by twenty, you are likely in one of the structured domains where it belongs. Re-running that comparison whenever you change embedding models keeps the answer current, and because a Chase Agents automation can pin its own model provider per step, dropping in a cheaper embedder to test against your current one is a configuration change rather than a rewrite.

Before your next RAG project reaches for the semantic splitter on principle, run the two-number comparison on your own corpus. If you cannot show the gain on your data, you are paying embedding costs to match what an 800-character recursive split would have done for free. Find out which case you are in before the bill arrives.