G raph QL Query Generation: A Large Training and Benchmarking Dataset (2024)

Abstract: GraphQL is a powerful query language for APIs that allows clients to fetch precise data efficiently and flexibly, querying multiple resources with a single request. However, crafting complex GraphQL query operations can be challenging. Large Language Models (LLMs) offer an alternative by generating GraphQL queries from natural language, but they struggle due to limited exposure to publicly available GraphQL schemas, often resulting in invalid or suboptimal queries. Furthermore, no benchmark test data suite is available to reliably evaluate the performance of contemporary LLMs.To address this, we present a large-scale, cross-domain Text-to-GraphQL query operation dataset. The dataset includes 10,940 training triples spanning 185 cross-source data stores and 957 test triples over 14 data stores. Each triple consists of a GraphQL schema, GraphQL query operation, and corresponding natural language query. The dataset has been predominantly manually created, with natural language paraphrasing, and carefully validated, requiring approximately 1200 person-hours. In our evaluation, we tested 10 state-of-the-art LLMs using our test dataset. The best-performing model achieved an accuracy of only around 50% with one in-context few-shot example, underscoring the necessity for custom fine-tuning. To support further research and benchmarking, we are releasing the training and test datasets under the MIT License. The dataset is available at https://github.com/stepzen-dev/NL2GQL.

## introduction graphql is a query language and runtime for apis, providing an efficient, powerful, and flexible alternative to rest apis. it allows users to request precise data across multiple sources in a single query, minimizing extraneous information and optimizing network resource usage. this makes graphql ideal for applications on resource-limited devices. figure 1 illustrates an exemplar graphql schema that enables users to retrieve student information. for clarity, the resolver functions that fetch the actual data have been omitted. however, in deployment, data for type student is fetched from a relational database, while their personal details are retrieved via a secure api endpoint. this illustrates how graphql can integrate disparate data sources into a single coherent interface for efficient data retrieval. it is crucial to note that this schema provides only two access points for data retrieval: 1) stu-dentlist, which fetches details of all students, and 2) get_students, which filters students based on a weight predicate (less than). all permissible graphql query operations for this schema can utilize only these two access points. for example, the query in figure 2a retrieves the name and address of all students, while figure 2b returns the names of students with weight less than 65. however, the query in figure 2c is invalid because the schema does not support a greater-than predicate for student weight. in general, challenges may also arise from inherent cyclic dependencies across type nodes, cross-source endpoints offering varying capabilities based on the data source, and custom schema extensions. this underscores the necessity for users to thoroughly understand their graphql schema to construct valid query operations. an interesting alternative is to use state-of-theart llms to generate graphql query operations from natural language queries. however, due to the limited availability of publicly accessible graphql schemas, these llms have not been sufficiently exposed to graphql data during their training. consequently, they often struggle to generate valid and optimized graphql query operations. furthermore, there is currently no comprehensive test dataset available to benchmark the performance of llms in generating graphql operations. to remedy this, we present a large-scale, crossdomain text-to-graphql query operation dataset. the dataset includes 10,940 training data triples spanning 185 cross-source data stores and 957 test triples spanning 14 cross-source data stores. each data triple consists of a graphql schema, graphql query operation, and the corresponding natural language query. to facilitate this task, we designed and implemented a custom data generation pipeline along with a web-based user interface for the review and annotation of each data element. this process involved six researchers and required approximately 1,200 person-hours for data generation and validation. next, we developed an evaluation pipeline that takes as input the graphql schema, the groundtruth graphql query operation, and the llmgenerated operation to check for query equivalence. this pipeline was used to validate the effectiveness of our test dataset. using this pipeline, we assessed the performance of 10 state-of-the-art llms, and surprisingly, the best model achieved only approximately 50% accuracy with one in-context few-shot example. these results underscore the necessity of creating complex graphql datasets for further model fine-tuning and benchmarking. ## dataset creation in this section, we begin by outlining the assumptions, followed by a comprehensive overview of the methodology employed to generate the training and test datasets. the entire dataset creation process is summarized in figure 3 . ## experiments and results we conducted experiments with 10 state-of-theart llms, with parameter sizes ranging from 3b to 34b. we employed three experimental settings: zero-shot, one-shot, and two-shot, where 'shot' refers to the number of in-context examples provided. we used greedy decoding to generate a maximum of 500 tokens for all llms. this study aims to: (1) demonstrate that our test dataset poses a significant challenge to the llms (2) assess whether providing a few in-context examples improves the models' ability to generate graphql queries, and (3) establish initial performance benchmarks for the text-to-graphql query generation task. during the evaluation, we constructed prompts that included instructions summarizing the generation task and incorporated 0, 1, or 2 few-shot samples, depending on the setting. each prompt also included the test schema and the natural language query. a sample prompt is outlined in appendix d. after obtaining results from the llm, we executed both the generated graphql query and the ground truth query, and then compared their outputs to evaluate accuracy. for each of the three experimental settings-zero-shot, one-shot, and two-shot-we have compiled the results in tables 2 , 3 , and 4, respectively. in these tables, the first column displays the model name, while the next eight columns detail the performance of each model on individual categories of graphql queries. the penultimate column indicates the fraction of test cases that failed due to exceeding the available llm context length, and the final column summarizes the overall performance across all categories. this category-wise presentation of results highlights the models' capabilities in generating respective graphql queries, providing insights that could guide the selection of the most appropriate model for specific use cases. furthermore, as shown in introducing one or two in-context examples led to marginal performance improvements in some models, with the granite-34b-code-instruct llm achieving the highest accuracy at around 50%. however, this improvement remains insufficient for real-world industrial applications. models with lower context limits, such as flan-t5, faced length errors due to prompts exceeding their token limits. this issue represents a significant bottleneck for employing in-context learning for graphql query generation, as graphql schemas are typically large and could easily surpass context limits, preventing the inclusion of even a single example. thus, enhancing the base model's performance by tuning it with a comprehensive graphql dataset remains the only viable option in such cases. this underscores the critical need for specialized graphql datasets to support the research community. training datasets can be employed for finetuning, prompt-tuning, or enhancing prompt engineering techniques, all aimed at improving llm performance in graphql query generation tasks. a brief discussion on the utility of the training dataset is provided in appendix c. meanwhile, the test dataset will be utilized for benchmarking generation capabilities, thereby establishing a measure of confidence in the llm's ability to generate graphql queries. ## related work graphql has gained significant attention in both academia and industry. while there have been attempts to utilize llms for graphql query generation @xcite @xcite , to our knowledge, there is no formal study or available training or benchmarking datasets to improve and evaluate this capability. therefore, in this section, we briefly review the basic literature on graphql. studies comparing rest and graphql apis highlight several advantages of graphql. for instance, @xcite shows that graphql reduces client-server interactions and minimizes json payload sizes, while @xcite demonstrates that graphql queries are eas-ier to implement. additional studies, such as @xcite , further explore the benefits of graphql. beyond the advantages over rest apis, recent research has focused on testing graphql queries @xcite and conducting mapping investigations @xcite . graphql has also become a critical component in real-world applications (gq, 2024) and businesses @xcite . its potential for data access and integration across heterogeneous sources is shown in @xcite . furthermore, from an industry perspective, the adoption of graphql is expected to increase in the near future. according to a recent gartner report, 'by 2027, more than 60% of enterprises will use graphql in production, up from less than 30% in 2024' (gar, 2024). ## conclusion and future work in this study, we created and validated a comprehensive text-to-graphql query operation dataset to enhance and benchmark the performance of llms in generating precise graphql queries from natural language inputs. we employed two distinct methodologies for dataset creation, integrating both automated tools and manual query generation. this approach ensured the comprehensiveness and quality of the dataset, providing a robust resource for the enhancement and evaluation of llm capabilities for graphql query generation task. our team, composed of six researchers across two geographical locations, invested approximately 1,200 person-hours in the creation and validation of the dataset. to ensure careful and responsible curation, we developed a custom web-based user interface for detailed review and annotation of each data element. future work will include expanding the dataset to relax some of the assumptions and address the limitations highlighted in this paper. we believe that our dataset will significantly contribute to advancing research in graphql query generation and the practical application of llms in the real world.