Auto-Cypher: Improving LLM s on Cypher generation via LLM -supervised generation-verification framework (2025)

Abstract: Graph databases like Neo4j are gaining popularity for handling complex, interconnected data, over traditional relational databases in modeling and querying relationships. While translating natural language into SQL queries is well-researched, generating Cypher queries for Neo4j remains relatively underexplored. In this work, we present an automated, LLM Supervised, pipeline to generate high quality synthetic data for Text2Cypher. Our Cypher data generation pipeline introduces LLM-As-Database-Filler, a novel strategy for ensuring Cypher query correctness, thus resulting in high quality generations. Using our pipeline, we generate high quality Text2Cypher data - SynthCypher containing 29.8k instances across various domains and queries with varying complexities. Training open-source LLMs like LLaMa-3.1-8B, Mistral-7B, and QWEN7B on SynthCypher results in performance gains of up to 40% on the Text2Cypher test split and 30% on the SPIDER benchmark, adapted for graph databases.

## introduction as the use of graph databases like neo4j @xcite grows, converting natural language into cypher queries (text2cypher) is becoming increasingly important. cypher @xcite , designed for querying and analyzing graph data, is well-suited for applications such as social networks, recommendation systems, and knowledge graphs @xcite . however, generating the relationships: (energysource)-[:produces]->(utility) (utility)-[:uses]->(energysource) ... the relationships: (:threat)-[:exploits]->(:vulnerability) (:vulnerability)-[:causes]->(:incident) (:incident)-[:generates]->(:alert) (:alert)-[:affects]->(:asset) … ## related work prior works on natural language querying of knowledge graphs using cypher has mostly focused on traditional ner based extraction @xcite or manual annotation @xcite approaches which make them both limited in scope, and cumbersome to write. llms have shown promising potential for text2cypher task where recently, neo4j labs published a gpt-4o generated dataset (tom, 2024), initiating first efforts on text2cypher data generation. importantly, this text2cypher data without any validation steps on a limited domain set, with only 6 query types on huggingface @xcite results only in 50% correctly executable cyphers. concurrent (peer-reviewed unpublished) synth2c @xcite generates cyphers using gpt-4o similar to neo4j labs as well as a templatized pipeline with traditional nlp techniques and llm-as-judge to validate generated cypher descriptions against original questions. however, this technique again does not check for execution correctness and is furthermore limited only to medical domain (with datasets not publicly available). the text2sql problem has been extensively studied in the literature, with numerous benchmarks and datasets @xcite @xcite @xcite . among these, spi-der @xcite is specially a prominent dataset covering a broad range of real-world scenarios. however, its real-world applicability remains uncertain, as evidenced by the spider-v2 @xcite benchmark, where gpt-4 achieves only a 6% pass@1. ## data generation pipeline synthetic data generation @xcite @xcite have proven highly effective. we use llms such as llama 3.1 70b @xcite , mixtral 8x22b @xcite , and gpt-4 (openai, 2023) to automatically generate diverse domains, schemas, natural language queries, and cypher queries. our pipeline covers a broad range of domains and query types, ensuring diversity across topics and difficulty. from data generation to validation, all steps are autonomously managed by models and scripts, allowing the process to run at scale. generated cypher queries are executed and validated against expected results to ensure quality. step 1: schema generation: we begin by random selection of the seed domains (e.g., e-commerce, inventory management) from neo4j (neo, 2024) example databases. we then use mixtral to expand these domains to cover 700 distinct domains. a skeleton schema is generated for each domain, outlining the nodes and relationships (block 1 in figure 2 ). these schemas are validated with gpt-4 for correctness and manually reviewed for coherence and real-world utility in 25% of cases. see appendix b for more details on schema generation. step 2: natural language question and ground truth generation for each schema, we generate questions based on 109 predefined query types, such as "simple retrieval" or "sub-graph queries" (block 2 in figure 2 ). a dummy ground truth answer for each query is also generated. in the next stage, we fill the database with entries including this dummy answer as the right answer for the step 3: neo4j database population an empty neo4j database for each question is created which is populated with synthetic data that fits the schema, question, and ground truth. python-based code, generated by gpt-4, is used to create and populate the database with nodes, relationships, and data, ensuring consistency between the schema and ground truth (block 3 in figure 2 ). to the best of our knowledge, this strategy of filling the database conditioned on a arbitrarily chosen dummy ground truth has not been explored in literature before. reverse filling the database in this way enables execution of cypher queries to check for execution success and cypher-code correctness. appendix e(8,9) provides more details on the data population process. step 4: cypher query generation next, the llm generates cypher queries for each question (block 4 in figure 2 ). following latest work in inference time scaling, we allow the llm to amply reason through various aspects of the cypher query, such as relevant nodes, relationships, properties, nuances of the question as well as best coding practices. this iterative chain-of-thought reasoning process coupled with execution checks against the synthetically filled database ensures only the highest quality data is generated. see appendix f for details on query generation. ## results as shown in table-2, our synthcypher dataset leads to significant improvements on both benchmarks across models. we draw several key observations: ## conclusion in this work, we highlight and address the text2cypher gap in current open source models, and introduced a novel pipeline to automatically generate and validate high quality text2cypher data. our presented dataset synthcypher from our pipeline leads to substantial performance improvements across multiple llms. we also provide two evaluation benchmarks for future works in this direction.