ZOGRASCOPE : A New Benchmark for Semantic Parsing over Property Graphs (2025)

Abstract: In recent years, the need for natural language interfaces to knowledge graphs has become increasingly important since they enable easy and efficient access to the information contained in them. In particular, property graphs (PGs) have seen increased adoption as a means of representing complex structured information. Despite their growing popularity in industry, PGs remain relatively underrepresented in semantic parsing research with a lack of resources for evaluation. To address this gap, we introduce ZOGRASCOPE, a benchmark designed specifically for PGs and queries written in Cypher. Our benchmark includes a diverse set of manually annotated queries of varying complexity and is organized into three partitions: iid, compositional and length. We complement this paper with a set of experiments that test the performance of different LLMs in a variety of learning settings.

## introduction knowledge graphs (kgs) have become increasingly prominent for representing complex relations between entities across domains and several graph data models have been proposed. accessing the information contained in a pg typically requires significant expertise in query languages such as cypher @xcite or gremlin @xcite . to reduce this barrier, natural language interfaces to kgs have emerged as a critical tool. this has driven research in semantic parsing and knowledge-based question answering, which aims to translate natural language questions into executable graph queries @xcite @xcite @xcite . despite advances, the task remains challenging due to the inherent complexity of natural language and the variability of graph schemas @xcite . annotated data is essential for advancing development in the area, but acquiring it is both ex-pensive and labor-intensive. besides, most available benchmarks have focused primarily on rdfstyle graphs. unlike rdf, property graph nodes can have an arbitrary list of associated properties enabling the specification of rich and semantically intuitive data models. pgs clearly distinguish between entity properties and entity relations, whereas rdf graphs conflate the two concepts. resources for pgs remain severely limited; many available cypher benchmarks are either synthetically generated without supervision or adapted from non-property graph sources. to address these challenges, we introduce zo-grascope, a benchmark designed for property graphs built from an open-access crime investigation graph using cypher queries. the dataset consists of annotated pairs of natural language questions and cypher queries. unlike most prior work, each example is manually annotated by experts. llms were used only for intermediate paraphrasing to enhance lexical variety, with all outputs manually verified for correctness. zograscope comprises 5k samples, featuring a range of query types, and includes iid, compositional and length train/test partitions. in addition, the paper includes a wide range of experiments to asses llms performance in a variety of learning settings. we hope that zograscope will serve as a valuable resource for advancing research in semantic parsing over property graphs. we release zograscope at https:// github.com/interact-erc/zograscope . ## related work few benchmarks exist for evaluating semantic parsers on property graphs, most of which are adapted from rdf and tabular datasets, or synthetically generated by llms. using the first approach @xcite and tiwari et al. (2025) adapt sql benchmarks. feng et al. (2025) and nie query match ( x0 : person ) -[: knows ] -( x1 : person ) -[: family_rel ] -( x2 : person where x2 . @xmath0. surname . value ]") return x0 . nhs_no proto-nl list all national health service number of people who knows people that has a family relation with people with last name that is [x2.surname.value] human annotation list all national health service numbers of people who knows someone that is family related to a person with surname [x2.surname.value] ## dataset creation the zograscope benchmark is based on the publicly available pole graph foot_0 , which contains a total of 61, 521 nodes and 105, 840 edges. it includes 11 entity classes, 32 unique properties, and 17 relation types. the graph focuses on crime investigations, modeling the relationships between people, objects, locations and events. for instance, a query might ask for the name of an officer investigating a particular crime that occurred at a specific location and involved a particular vehicle. creating the benchmark involves a five step process: 1) we automatically generate variable anonymized cypher queries. 2) each anonymized cypher query is manually annotated by an expert with a natural language realization. 3) for each query we instantiate the values of the anonymized variables in both the cypher query and its natural language realization. 4) an llm is used to suggest paraphrases of the instantiated natural language realizations and 5) a human annotator verifies and corrects the instantiated paraphrases. nl what is the number of shoplifting investigations conducted by police constables? query match ( x0 : crime where x0 . @xmath1: investigated_by ] -( x1 : officer where x1 . @xmath2.crime.type:@xmath3.officer.rank:police @xmath4: phonecall ) -[: caller ] -( x1 : phone ) -[: has_phone ] -( x2 : person ) -[: party_to ] -( x3 : crime where x3 . @xmath5. call_duration order by x0 . call_date desc limit 1 extra info x3.crime.date:11/08/2017 = 11/08/2017 nl can you list those connected to a crime handled by officer rockall, with the investigation finished and no suspect found? query match ( x0 : person ) -[: party_to ] -( x1 : crime where x1 . @xmath6; no suspect identified ") -[: investigated_by ] -( x2 : officer where x2 . @xmath7.crime.last_outcome:investigation complete; no suspect @xmath8.officer.surname:@xmath9: total number of samples in each partition and their distribution among different query lengths. for each query, we also associate a determinis-tically generated proto-natural language sentence realization, this is only used to aid the human annotator who can decide to ignore it if he prefers to only look at the cypher query. the annotator is then asked to generate a realistic natural language realization of the query. annotators are provided with the graph schema, which includes information on entity classes, properties, and relations. the result of this step is an anonymized cypher query and its corresponding anonymized natural language realization. we then instantiate the anonymized variables with valid values from the graph, making sure that the query yields non-empty results. we generate three instantiations per query. subsequently, to enrich the lexical diversity of the natural language queries we use an llm (gpt4o) to produce three paraphrases for each instantiation. in the final step a human annotator reviews each paraphrase verifying and correcting them if needed. the final dataset generated using this approach has around 5k samples and the query lengths range from 2 to 5 nodes. we show some examples in ## experiments we ran a series of experiments to examine semantic parsing performance on zograscope. specifically, we assess different transformer-based models under four distinct settings: note that for the last three settings we include in the llm prompt the schema description of the graph with information about classes, properties and relations (see appendix b). we present the results of these evaluations using a range of llms including small models that can run locally: llama-3.2-3b @xcite , qwen3-4b @xcite , and mistral-7b-v0.1 @xcite as well as a larger proprietary model gpt-4o: (gpt-4o-2024-08-06). for the evaluation metric, we use execution accuracy, where a prediction is deemed correct if it yields identical results to the reference query when executed on the knowledge graph. since we are interested in the semantic parsing task, we run these experiments assuming perfect entity linking by passing the gold entities to the models. ## conclusions in this paper, we introduced zograscope, a novel benchmark designed to evaluate the performance of semantic parsers over cypher property graphs. built on the pole graph, which models crime investigations, the benchmark includes a set of cypher queries paired with natural language realizations, manually annotated by expert annotators. we hope that zograscope will contribute to advancing research in automatic text-to-cypher translation, addressing the limited availability of similar resources and fostering progress in the field.