How Much Reasoning Do Retrieval-Augmented Models Add beyond LLMs? A Benchmarking Framework for Multi-Hop Inference over Hybrid Knowledge

Modern LLM systems increasingly rely on retrieval-augmented generation (RAG) and knowledge-graph-augmented reasoning (KG-RAG) to handle knowledge-intensive tasks. But how much reasoning do these systems truly add beyond the base LLM’s parametric knowledge?

We introduce HybridRAG-Bench, a contamination-aware benchmarking framework for evaluating retrieval-intensive, multi-hop reasoning over hybrid knowledge (text + structured graphs). Rather than releasing a static dataset, we provide a reusable benchmark construction pipeline that generates challenging, reasoning-grounded QA pairs from time-scoped scientific corpora.

😆 Takeaways:

1️⃣ Benchmark contamination hides true reasoning gaps.
Many widely used QA benchmarks overlap heavily with LLM pretraining data, making it difficult to distinguish retrieval-based reasoning from parametric recall. When evaluated on temporally controlled, contamination-aware data, LLM-only performance drops substantially.

2️⃣ Retrieval alone is not enough.
Text-based RAG improves over LLM-only prompting, but struggles on structured multi-hop and relational questions. Naïve KG injection can even hurt performance due to noise.

3️⃣ Structured knowledge adds measurable reasoning benefits.
Hybrid methods that combine graph traversal with textual retrieval consistently outperform text-only RAG—especially on difficult multi-hop and compositional reasoning tasks.

4️⃣ Fine-grained question types reveal failure modes.
By breaking evaluation into single-hop, multi-hop, difficult multi-hop, counterfactual, and open-ended questions, HybridRAG-Bench surfaces distinct weaknesses in retrieval, evidence integration, and reasoning strategies that aggregate accuracy hides.

5️⃣ Reasoning gains are method-dependent—not just scale-dependent.
The performance gap between different retrieval/reasoning strategies is often larger than the gap from scaling model size alone, suggesting that system design matters as much as model scale.