When to Use Graphs in RAG

Despite GraphRAG's conceptual promise, recent studies have shown that it frequently underperforms vanilla vector RAG on many real-world tasks. The ICLR 2026 paper "When to Use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation" by Xiang et al. tackles this contradiction head-on, introducing GraphRAG-Bench to systematically determine where graph structures provide measurable benefits.

The Central Question

Is GraphRAG really effective, and in which scenarios do graph structures provide measurable improvements over standard vector retrieval? The paper's contribution is a benchmark designed to answer this precisely: GraphRAG-Bench evaluates models across four task categories of increasing difficulty, spanning fact retrieval, complex reasoning, contextual summarisation, and creative generation.

Task Category	Example	Graph RAG Benefit
Fact retrieval	"Who founded company X?"	Minimal — single-hop facts are well-served by vector search
Multi-hop reasoning	"Which products are affected by regulation Y via supplier Z?"	Significant — graph traversal captures path dependencies
Contextual summarisation	"Summarise the compliance exposure across all EU subsidiaries"	High — community summaries aggregate structured relationships
Creative generation	"Draft a risk report based on the current threat landscape"	Moderate — graph provides structured grounding

When GraphRAG Wins

GraphRAG consistently outperforms vanilla RAG on tasks where the answer depends on relationships spanning multiple entities, documents, or business rules. The paper identifies three conditions that predict GraphRAG success:

1. Multi-hop dependency. The answer requires traversing two or more relationship steps (e.g., supplier → component → product → regulation).

2. Entity centrality. The question revolves around how entities connect rather than what individual documents contain. Questions about compliance exposure, supply chain risk, and organisational impact all benefit.

3. Hierarchical structure. The data has a natural hierarchy (organisation charts, product categories, regulatory frameworks) that vector similarity cannot capture.

Benchmark Dimension	GraphRAG	Vanilla RAG	Improvement
Single-hop fact retrieval	91.2%	93.1%	−1.9 pp
Multi-hop reasoning (2 hops)	84.7%	72.3%	+12.4 pp
Multi-hop reasoning (3+ hops)	71.5%	48.9%	+22.6 pp
Contextual summarisation	88.3%	76.1%	+12.2 pp
Creative generation	79.8%	74.2%	+5.6 pp

The headline finding: GraphRAG underperforms on simple factual lookups but delivers a 12–23 percentage point improvement on multi-hop and relational tasks.

When GraphRAG Underperforms

On fact retrieval tasks where the answer is contained in a single document or passage, the overhead of graph construction and traversal is pure cost with no benefit. The paper finds that vanilla vector RAG is faster and equally accurate for:

• Simple Q&A where the answer appears verbatim in a source document
• Keyword-based lookup tasks
• Questions where the relevant context fits within a single retrieval window

Practical Guidelines

Based on the GraphRAG-Bench findings, the paper offers three decision rules for enterprise teams:

1. Use GraphRAG when the question domain is relational by nature — compliance, supply chain, organisational structure, or any regulated workflow where the evidence path must be traceable.

2. Use vanilla RAG when the question is a simple lookup — document Q&A, policy search, or any task where the answer resides in one or two passages.

3. Use hybrid GraphRAG (graph + vector) when the task mix is unpredictable — a general-purpose enterprise search that must handle both simple lookups and complex relational queries.

Enterprise Implications

The paper's most valuable contribution for enterprise teams is not a new architecture but a validated decision framework. GraphRAG is not universally superior; it is conditionally superior in precisely defined scenarios. Teams evaluating GraphRAG should start with a workload audit against the paper's three conditions, adopt GraphRAG only for the tasks that satisfy them, and leave simple retrieval on vector search. This targeted approach maximises the ROI of graph infrastructure while avoiding the cost of over-engineering.

The full benchmark, dataset, and evaluation pipeline are available at github.com/GraphRAG-Bench/GraphRAG-Benchmark.