Deploying GraphRAG in Production: Architecture, Cost, and Operations

The Gap Between Demo and Production

GraphRAG demos are compelling. Show a system that answers "How does this API change affect the three downstream services that depend on it?" — a multi-hop question that vector search cannot answer — and stakeholders are convinced. The hard part comes after the demo: turning a Jupyter notebook into a system that runs reliably at 100 queries per second, costs a predictable amount to operate, and stays fresh as documents change.

By mid-2026, enough GraphRAG systems have been deployed in production to separate myth from pattern. This article synthesises real deployment lessons into a practical guide covering architecture decisions, cost modelling, incremental indexing, query reliability, and the operational practices that keep GraphRAG systems healthy.

When GraphRAG Earns Its Keep

The first decision in any production deployment is when to use GraphRAG at all. Every deployment that has scaled successfully applies GraphRAG as an additive backend, not a wholesale replacement for vector RAG.

Query Type	Recommended Method	Why
Simple fact retrieval	Vector RAG	Cheaper, faster, no graph overhead
Single-document Q&A	Vector RAG	Dense retrieval is sufficient
Multi-hop (2–3 hops)	GraphRAG + vector hybrid	Graph traversal narrows candidates, vectors fill context
Cross-document synthesis	GraphRAG (global mode)	Community summaries aggregate across documents
Relational queries ("which X depends on Y?")	GraphRAG (template Cypher)	Graph traversal is the only reliable approach
High-frequency factual lookups	Cache-Augmented Generation	No retrieval needed for known patterns

The well-documented rule from production deployments: roughly 7% of queries genuinely require graph traversal. For the other 93%, vector RAG with a good reranker is faster and cheaper. A routing classifier — a small LLM or a traditional classifier trained on query logs — directs each query to the appropriate pipeline.

Three Architectural Patterns That Survive Production

After a year of production GraphRAG adoption, three architectural patterns recur across successful deployments.

Pattern 1: Hybrid Vector + Graph

The most common pattern. A routing layer decides per-query whether to activate graph traversal or stay in vector space. The graph and the vector index operate on the same corpus but serve different query profiles.

                     ┌─────────────────┐
                     │  Query Router   │
                     │  (Classifier)   │
                     └──┬──────────┬───┘
                        │          │
                   ┌────▼──┐  ┌────▼────┐
                   │Vector │  │  Graph  │
                   │Retrie.│  │Traversal│
                   │(Qdrant│  │ (Neo4j) │
                   │/pgvec)│  │         │
                   └───┬───┘  └────┬────┘
                        │          │
                        └────┬─────┘
                        ┌────▼─────┐
                        │ Reranker │
                        │ (Cross-  │
                        │ encoder) │
                        └────┬─────┘
                        ┌────▼─────┐
                        │  LLM     │
                        │ Synthesis│
                        └──────────┘

The reranker is critical: it merges graph traversal results (entity IDs, relationship paths) with vector search results (document chunks) into a unified context window, deduplicating and scoring by relevance. Production systems using a cross-encoder reranker in this fusion topology report 34% higher answer accuracy than vector-only retrieval while maintaining p95 latency under one second for 90% of queries.

Pattern 2: Layered Extraction

Extraction cost — the LLM calls needed to turn documents into entities, relationships, and community summaries — dominates GraphRAG operating budgets. A layered extraction pattern uses a small, cheap model (a 1–3B parameter model) for bulk entity extraction, and reserves a frontier model only for community summarisation and high-value relationship inference.

# Layered extraction: small model for bulk, frontier model for synthesis
from neo4j_graphrag.experimental.pipeline.kg_builder import SimpleKGPipeline
from neo4j_graphrag.llm import OpenAILLM

# Small model for entity/relation extraction
extraction_llm = OpenAILLM(model_name="gpt-4o-mini", max_tokens=2000)
# Frontier model for community summarisation
summary_llm = OpenAILLM(model_name="gpt-4o", max_tokens=8000)

pipeline = SimpleKGPipeline(
    driver=driver,
    llm=extraction_llm,
    embedder=embedder,
    entities=["Customer", "Order", "Product", "Supplier"],
    relations=["PLACED", "CONTAINS", "SOURCED_FROM"],
    enforce_schema="STRICT"
)

The saving is roughly an order of magnitude on extraction cost. Community summarisation remains the most expensive single stage — but it runs on the aggregated graph, not on every document, so its cost scales with graph complexity rather than corpus size.

Pattern 3: Federated Graph

In enterprise environments where different departments own different data domains, a single monolithic graph creates coordination bottlenecks. The federated pattern lets each domain maintain its own graph with its own schema, plus an upper graph of common entities that bridges them.

This pattern survives reorganisations without full re-ingestion: when one department restructures its ontology, only its graph needs rebuilding. The bridging graph handles cross-domain queries by mapping entities to a shared vocabulary.

Storage: Neo4j or PostgreSQL?

One of the most debated decisions in GraphRAP production deployments is the storage backend. The conventional choice is Neo4j, but PostgreSQL — particularly with pgvector and the AGE extension — has gained significant ground.

Criterion	Neo4j	PostgreSQL + pgvector	PostgreSQL + AGE
Native graph traversal	✓	✗ (app-level joins)	✓ (Cypher via AGE)
Vector index	✓ (HNSW in GDS)	✓ (HNSW/IVF via pgvector)	✗ (use pgvector alongside)
Multi-hop query latency	~45ms p50	~200ms p50 (app joins)	~80ms p50
Ops complexity	Additional DB to manage	Familiar to most teams	Familiar + graph extension
Scaling cost (1K QPS, monthly)	~$4,800 (6 replicas)	~$1,800	~$2,400
Community detection	✓ (GDS: Leiden, Louvain)	✗	✗ (limited)

For mid-size graphs under a million entities, PostgreSQL with well-tuned indexes is operationally sensible and economically attractive. You avoid adding another database to your inventory, and many teams already run PostgreSQL in production. The trade-off is that deep multi-hop traversals (4+ hops) become expensive because they require application-level joins rather than constant-time pointer chasing.

For graphs exceeding a million entities, or when you need pathfinding algorithms (shortest path, PageRank, community detection), Neo4j's native graph engine and the Graph Data Science library justify the additional operational cost. As covered in Graph Algorithms in Production with Neo4j GDS, GDS provides over 50 in-memory algorithms that cannot be replicated in PostgreSQL without significant engineering effort.

Cost Modelling: Know Where the Money Goes

The single biggest surprise for teams deploying GraphRAG in production is the 10–30x ingestion cost premium over vectorising the same corpus. LLM calls for entity extraction, relationship inference, and community summarisation dominate the budget. In representative production runs, these two stages account for approximately 90% of total build cost and runtime.

Build a cost model before you deploy. Run one full indexing pass on a representative corpus sample, then extrapolate:

Stage	Cost Driver	Optimisation Levers
Document chunking	Negligible	—
Entity extraction	LLM tokens per document	Use small model; trim prompts; batch with async I/O
Relationship inference	LLM tokens per chunk pair	Dependency parsing as first pass; LLM only for low-confidence edges
Embedding generation	API cost per chunk	Negligible compared to LLM stages
Community detection	Free (Leiden in GDS)	—
Community summarisation	LLM tokens per community	Most expensive single stage; limit to top-N communities
Query-time LLM call	Tokens per query	Cache frequent queries; use smaller synthesis model

The practical optimisation hierarchy:

1. Use a small model for extraction — gpt-4o-mini or a local 7B model for 90% of extraction, frontier model only for summarisation
2. Trim prompts aggressively — remove examples that apply only to a subset of documents; measure token usage per stage
3. Set explicit rate limits — TPM/RPM limits reduce 429 retries; bounded retries prevent workflow hangs
4. Tune community resolution — larger communities mean fewer summaries but coarser granularity; the optimal setting is corpus-dependent

Query Reliability: Templates, Not Free Generation

The most common production failure in GraphRAG is unreliable Cypher generation. Letting an LLM write Cypher queries directly produces correct results roughly 77% of the time — impressive for a demo, unacceptable for production.

The proven alternative is a template-based approach:

1. A query understanding stage classifies the natural-language question into one of ~30 query patterns
2. Each pattern has a parameterised Cypher template
3. The LLM fills in parameters (entity names, filters, aggregation criteria) rather than writing Cypher from scratch

-- Template for: "Which [target] are connected to [source] via [path]?"
MATCH (source:{{source_label}} {name: $source_name})
{{path_template}}
RETURN DISTINCT target.name AS result
LIMIT $limit

This approach drops query errors from 23% to under 4%. The trade-off is flexibility — new query patterns require new templates. Production systems maintain roughly 30 templates and add two to three new ones per month based on query log analysis.

Additional reliability measures:

• Traversal depth limits: default of 3 hops, maximum of 5 for explicitly analytical queries
• Result count caps: maximum of 500 result nodes per query
• Timeouts and budgets: when a traversal exceeds budget, return partial results with an explanation rather than timing out silently
• Schema versioning: version-controlled Cypher templates with regression test suites

Incremental Indexing and Freshness

Full re-indexing of a production knowledge graph is expensive. Incremental indexing — processing only document deltas — is essential for operational viability.

The key insight from production deployments is that freshness in a knowledge graph is not just about keeping data current. In systems where users ask "What changed?" or "Why does this behave differently now?", the graph must be designed so that evolution is visible and traceable, not just overwritten.

# Incremental update: process only new and modified documents
def incremental_index(pipeline, last_run_timestamp: datetime):
    new_docs = fetch_documents(updated_since=last_run_timestamp)
    for doc_batch in batched(new_docs, size=50):
        asyncio.run(pipeline.run_async(doc_batch))
    # Recompute only affected communities
    recompute_communities(affected_entity_ids=detect_changed_entities())

An explicit operational policy should specify:

• When to do incremental updates: for small, non-structural changes
• When to do full re-indexing: when detecting major schema changes or drift beyond a threshold
• How to handle conflicts: when the same entity is updated by concurrent extraction jobs

Community structure is particularly sensitive to incremental updates. Most production systems use a hybrid approach: schedule community re-detection on affected subgraphs weekly, while preserving stable community IDs elsewhere to maintain query consistency.

Observability Beyond Vector RAG

GraphRAG requires observability dimensions that vector-only RAG does not. If you are monitoring GraphRAG with the same metrics you use for vector RAG, you are blind to the failure modes that matter.

Metric	Why It Matters	Alert Threshold
Graph traversal p50/p95	Deep traversals can explode without bounds	Alarm if p95 > 2s
Community selection accuracy	Wrong community → irrelevant context	Sample 100 queries weekly
Edge confidence distribution	Low-confidence edges degrade quality	Flag edges < 0.5 confidence
Stale-node ratio	Outdated entities mislead queries	Rebuild if > 10% stale
Cache hit rate (community summaries)	Repeated LLM calls waste budget	Optimise if < 60%
Query routing accuracy	Wrong pipeline → wrong answer	Monitor via A/B comparison

Semantic Caching

Semantic caching is more important in GraphRAG than in basic RAG because many queries repeatedly touch the same high-value entities, communities, and summaries. Cache not just final answers but also intermediate artifacts: normalised entities, extracted triples, community summaries, and graph traversal results.

Cache invalidation policies must account for graph drift. Production systems typically implement time-based expiration combined with change detection on community membership: when a community grows or shrinks beyond a threshold, its cached summary is marked stale and regenerated on the next access.

Reference Architecture

A production GraphRAG deployment that embodies these patterns looks like this:

Layer	Component	Purpose
Ingestion	Async extraction pipeline (small LLM)	Entity and relationship extraction from source documents
Storage	Neo4j (entity graph) + pgvector (dense vectors)	Graph traversal and semantic search
Indexing	GDS Leiden community detection	Hierarchical community partitioning
Query classification	Lightweight classifier (~30 templates)	Route query to graph, vector, or cache
Retrieval	Template-based Cypher + vector HNSW	Reliable graph traversal and semantic search
Fusion	Cross-encoder reranker	Merge and rank graph + vector results
Generation	LLM synthesis (small model for most queries)	Context-grounded answer generation
Cache	Redis (semantic + summary cache)	Reduce repeated LLM calls
Observability	OpenTelemetry + custom GraphRAG metrics	Traversal depth, routing accuracy, cost per query
CI/CD	Versioned Cypher templates + eval suite	Regression-free schema and query updates

Scale this architecture incrementally. Start with a single Neo4j instance and a vector index, prove value on a bounded domain, then expand. The teams that succeed with GraphRAG in production are the ones that treat it as an operational platform from day one — with explicit cost budgets, update cadences, monitoring, and rollback plans — rather than as a research project that somehow escaped the notebook.

For a deeper look at the GraphRAG variants available today, see GraphRAG Variants Compared in 2026. For a cost-efficient alternative, read Lazy GraphRAG on $30.