Your RAG pipeline retrieves the right documents. The LLM ignores them anyway.

LLM & Model Guides · By Caleb Sakala · April 25, 2026

You checked the chunks. You logged the retrieval scores. Cosine similarity sits above 0.85, and the top-three results contain exactly the paragraph the user asked about. The answer is still wrong. This is where most guides tell you to tweak chunk sizes or swap embedding models. But when your RAG pipeline is not working despite correct retrieval, the failure lives in generation, and the fixes look nothing like the standard troubleshooting checklist.

The lost-in-the-middle effect kills answers that good retrieval delivered

Nelson Liu and collaborators at Stanford and the University of Washington published “Lost in the Middle” in 2023 (later in TACL 2024). Their finding: LLMs attend most strongly to information at the very beginning and very end of the context window. Information placed in the middle gets partially ignored. The performance degradation follows a U-shaped curve, and it held across context windows from 4K to 32K tokens.

A retrieval system can return five perfect chunks, but if the most relevant chunk lands at position three in the concatenated context, the model draws more heavily from chunks one and five. The answer sounds confident. It cites real-sounding details. Those details come from the wrong chunks.

# Default concatenation buries the best chunk in the middle
context = "\n\n".join(retrieved_chunks)

# Fix: reorder so highest-relevance chunk is first, second-best is last
ranked = sorted(retrieved_chunks, key=lambda c: c.score, reverse=True)
context = "\n\n".join([ranked[0]] + ranked[2:] + [ranked[1]])

Anthropic’s prompt engineering documentation recommends placing the most relevant context near the top of the prompt for long-context tasks. The reordering takes four lines of code. Most teams never apply it because nobody surfaces the chunk order between retrieval and generation.

LLMs override retrieved context with what they already “know”

Research presented at ECIR 2025 confirms that LLMs exhibit confirmation bias toward their parametric knowledge. When retrieved context contradicts something the model learned during training, the model often ignores the context and answers from memory instead.

This hits domain-specific RAG systems hardest. A company knowledge base that says “purchase orders above $500 require VP approval” will get overridden if the model’s training data contains thousands of examples citing $1,000 thresholds. The answer looks plausible because it IS a real threshold. Just not YOUR threshold.

# Test for parametric knowledge override
test_cases = [
    {"query": "What is the PO approval threshold?",
     "expected": "$500",  # from internal policy
     "common_answer": "$1,000"},  # from training data
    # Add 19 more cases where internal docs differ from public info
]

override_count = sum(
    1 for case in test_cases
    if case["common_answer"] in rag_pipeline.query(case["query"])
    and case["expected"] not in rag_pipeline.query(case["query"])
)
print(f"Override rate: {override_count / len(test_cases):.0%}")

There is no universal threshold for what “too much override” looks like. A medical knowledge base with standard clinical guidelines sees less override than a company policy base full of internal acronyms and non-standard approval flows. The number depends on how much your internal documents diverge from public information. But the test itself takes an afternoon to build, and the results are genuinely surprising more often than not.

The fix is explicit instruction in the system prompt: tell the model to prioritize the provided documents over its own knowledge, and to respond with “not found in provided documents” rather than guessing. Research from the FaithEval benchmark (4,900 test questions across faithfulness scenarios) found that explicit context-priority instructions measurably improve faithfulness scores across all tested models and configurations. The instruction change is one line in the system prompt, and it moved faithfulness scores across all tested model configurations in that benchmark.

How to tell if your RAG pipeline is not working at retrieval or generation

Pull 10 recent wrong answers. For each one, check: did retrieval return a relevant chunk? If yes, the problem is generation. Check whether the relevant chunk sat in the middle third of the context (lost-in-the-middle) or whether the wrong answer matches commonly known information rather than your documents (parametric override). Both can co-occur.

Retrieve less, not more

Here is the recommendation that most RAG optimization guides avoid giving: cut the number of retrieved chunks.

Teams default to retrieving 5, 10, sometimes 20 chunks because more context feels safer. The opposite is true in practice. Every irrelevant chunk competes for the model’s attention budget. With 20 chunks in context, even if 3 are relevant, they share attention with 17 distractors. The signal-to-noise ratio drops, and the lost-in-the-middle problem compounds because relevant chunks have more positions where they can land in the dead zone.

LangChain and similar frameworks often blur retrieval and generation into a single chain call. Nobody notices when retrieval returns 15 chunks instead of 3 because the intermediate state between those steps is invisible. Chase Agents handles this boundary differently: each automation step declares an input schema, so a retrieval step feeding a generation step has an explicit typed contract. If retrieval returns chunks with confidence scores below a defined threshold, the automation routes to a fallback path instead of feeding noise into the LLM.

A reranker that scores and filters chunks before they reach the model often matters more than upgrading your embedding model. Cohere’s Rerank API and open-source alternatives like ColBERTv2 both work here. The reranker controls what the model sees. Context quality is a curation problem.

The same principle extends to how automation workflows handle the boundary between retrieval and generation. Chase Agents’ action-type routing lets each step output typed results that the next step validates before proceeding. A document-processing workflow can check chunk confidence scores after retrieval and before generation, routing low-confidence queries to a human review queue rather than producing a confident wrong answer. Most RAG frameworks skip this validation entirely because the intermediate state is opaque, which is how teams end up debugging retrieval when the real failure happened one step later.

The question nobody asks when their RAG pipeline is not working

Go back to those 10 wrong answers. Count how many had correct retrieval. If the number is higher than you expected, the articles about chunk sizes and embedding dimensions are solving a problem you don’t have. Your LLM had the right information and chose not to use it. Until you fix generation, better retrieval will keep delivering the same wrong answers at higher confidence. What percentage of your RAG failures are retrieval failures, and what percentage are generation failures? If you have never measured that split, you have been debugging blind.