Guarding LLM-Generated Cypher: Testing Graph Queries in CI

Text-to-Cypher is the feature every graph product wants and every query engineer quietly fears. The LLM is excellent at producing Cypher that reads like the answer. It is equally good at producing a query that is wrong, catastrophically slow, or — in the worst case — a full-graph scan disguised as a question. The fix is not to abandon text-to-Cypher. It is to treat every generated query as untrusted input and move it behind the same discipline you apply to user-facing code: validation, ceilings, and tests in CI.

The three ways a generated query fails

1. Wrongness. Returns the wrong rows without erroring — the LLM misunderstood the schema or the question. Silent and the most dangerous.
2. Expense. Correct but expensive: an unanchored MATCH scans the whole graph, a missing index turns a lookup into a traversal.
3. Destruction. Malformed or adversarial input reaches MERGE, SET, or deletion clauses and mutates data.

Each needs a different guard.

Schema-aware validation before execution

The cheapest guard is structural: validate the generated Cypher against the schema before the database ever sees it.

• Freeze what it may touch. Only allow read clauses (MATCH, RETURN, WHERE, WITH, ORDER BY, limits); explicitly reject MERGE, SET, DETACH DELETE, and CALL unless the feature truly needs write.
• Bind to labels and properties you know exist. Parse the query and verify every :Label and property references a real one. A typo label is the classic "correct-looking, wrong" failure — catch it before the query.
• Reject unanchored full scans. A MATCH (n:Label) with no starting bound value and no path from a bound node is a scan; block it by default.

Result: a hard, fast, pre-execution filter that removes wrongness and the worst of the expense, with no query executed.

Query-cost ceilings as a second net

Schema validation stops wrong and structurally-bad queries. Cost ceilings catch the correct-but-expensive ones that slip through (index path choices, fan-out). Set explicit limits and enforce them:

• A hard node touch budget. Cap estimated/first-returned work (e.g., a bounded LIMIT enforced at the app layer, plus a max-returned-rows cap).
• Per-request timeout and row cap. A generated query that doesn't return within budget fails, not the database.
• A restricted API for generated queries. Route them through an executor that injects the budget and strips anything destructive, instead of passing them to a general-purpose driver.

Regression tests in CI against a fixture graph

The only thing that catches semantic wrongness reliably is running the queries against known data. Build a small, fast, isolated fixture graph that parodies your real schema, and test generated queries in CI:

1. Golden-answer tests. For a curated set of natural-language questions, run the generated Cypher against the fixture and assert the exact expected rows. Wrong rows → red.
2. Schema-contract tests. Assert that generated queries only use labels and properties that exist, and never contain write clauses.
3. Cost-bound tests. Assert the fixture queries touch fewer than N nodes / finish under a timeout — so a regression that turns $answers into scans fails the build.

Now text-to-Cypher has the same safety net as application code: a wrong or expensive query is caught by CI before it ships, instead of on production data.

The untrusted-input rule

The mental model that makes it work: a generated query is user input with extra steps. You would never pass raw user input straight to your graph driver; you validate, bind, and limit. Treat LLM output identically — it is just input you didn't type. Schema validation for what it may touch, ceilings for how much it may do, CI fixture tests for whether it's actually right.

Text-to-Cypher doesn't need to be trusted to be useful. It needs to be guarded. With validation, budgets, and a test harness, you publish the feature confidently — because every generated query has already proven it can't be wrong, can't blow up the graph, and can't do it silently twice.