Performance and Limitations of Fine-Tuned LLM s in SPARQL Query Generation (2025)

Abstract: Generative AI has simplified information access by enabling natural language-driven interactions between users and automated systems. In particular, Question Answering (QA) has emerged as a key application of AI, facilitating efficient access to complex information through dialogue systems and virtual assistants. The Large Language Models (LLMs) combined with Knowledge Graphs (KGs) have further enhanced QA systems, allowing them to not only correctly interpret natural language but also retrieve precise answers from structured data sources such as Wikidata and DBpedia. However, enabling LLMs to generate machine-readable SPARQL queries from natural language questions (NLQs) remains challenging, particularly for complex questions. In this study, we present experiments in fine-tuning LLMs for the task of NLQ-to-SPARQL transformation. We rely on benchmark datasets for training and testing the fine-tuned models, generating queries directly from questions written in English (without further processing of the input or output). By conducting an analytical study, we examine the effectiveness of each model, as well as the limitations associated with using fine-tuned LLMs to generate SPARQL.

## introduction in recent years, the interaction between users and automated systems across various domains has become the center of artificial intelligence (ai) research. one of the main challenges in this interaction is translating human-written questions into machine-readable formats. this is specifically true for knowledge graphs represented in rdf format. those contain vast amounts of information on a wide variety of topics, but would generally require a user to write a query in the sparql language to use them to answer specific questions. such queries provide accurate answers from reliable and structured data sources. however, they are accessible only to people with sparql knowledge and the time to formulate such queries. this is a significant barrier to the accessibility of information embedded in kgs. this study explores how llms, when fine-tuned, can generate accurate sparql queries from nlq, allowing direct human-friendly interaction with kgs such as dbpedia foot_0 and wikidata foot_1 , thus giving access to complicated systems to a wider range of people, including non-specialists. by investigating a range of llms, we aim to identify the conditions under which these models produce accurate sparql queries, while highlighting their limitations. these limitations include syntactic errors in generated queries, hallucinated identifiers, etc. our objective is to better understand the capabilities of different llms in this task, and to identify potential areas of improvement for future research. the remainder of this paper is organized as follows. in section 2, we present related works on qa and llm development. then, in section 3, we present our analytic study and the fine-tuning process of various meta's llama models to get a new model specialized on transforming nlq to sparql queries, called llama-kgqa (llama based model for knowledge graph question answering). section 4 presents our results, comparing the fine-tuned models with each other, as well as with other existing qa systems. finally, section 5 summarizes our findings and outlines future perspectives to enhance llama-kgqa's capabilities. ## related works question-answering is a branch of natural language processing (nlp) that aims to automatically respond to user questions asked in natural language. the goal of qa systems is to provide precise and contextually relevant answers to a wide range of questions by accessing various data sources, such as unstructured text (see, for example, @xcite ), structured databases (see, for example, (khanam and subbareddy, 2017)), or knowledge graphs (see, for example, @xcite ). qa systems have become central to many applications, including search engines, intelligent virtual assistants (e.g. siri, alexa, etc.), and customer support chatbots. qa systems are typically classified according to the type of data on which they rely to answer questions: text-based qa systems (tbqa) and knowledge-based qa systems (kbqa). the tbqa systems extract answers from large collections of unstructured or semi-structured text, such as documents, web pages, or research papers. they involve tasks such as document retrieval and answer extraction, relying heavily on nlp techniques such as information retrieval (see, for example, @xcite @xcite ), text classification (see, for example, @xcite ), and semantic matching (see, for example, @xcite ). kbqa systems utilize structured data sources, such as knowledge bases or knowledge graphs, where information is stored in a highly organized manner (such as rdf data). these systems interpret user queries given as nlq and translate them into formal queries (e.g. sparql for rdf-based knowledge graphs) that can directly retrieve factual answers from the knowledge base. in this paper, we focus specifically on kbqa systems that rely on knowledge graphs. those systems are discussed later in this section. large language models (llms) have significantly advanced the field of qa by enhancing the ability of ai assistance and chatbots to comprehend and generate natural language responses. llms such as gpt @xcite , bert @xcite , and meta-llama models (ai@meta, 2024) are pre-trained on massive datasets that include diverse text sources such as books, web pages, and scientific articles. this extensive pre-training enables llms to internalize vast amounts of general knowledge and linguistic structures, allowing them to respond to open-domain questions across various fields with minimal task-specific training. unlike traditional qa systems that rely on explicit query-to-answer mappings or structured knowledge bases, llms can generate nuanced, context-aware responses by leveraging their pre-trained language understanding models. however, llms also face challenges, such as actual generated errors or "hallucination" @xcite , where the models generate plausible but incorrect answers due to their reliance on learned patterns rather than factual verification. despite these challenges, llm-based qa systems are at the forefront of nlp, offering robust capabilities for applications in virtual assistants, search engines, and more. among the research questions that have arisen in recent years is the possibility for llms to efficiently generate machine-readable queries from questions written in natural language to interrogate information sources. for knowledge graphs, this involves transforming a question posed (for example) in english into a valid sparql queries to a specified kg @xcite . this could significantly improve linking human language with machine-readable data stores, such as kgs. this capability is crucial because kgs, such as dbpedia and wikidata, store vast amounts of structured information that can be accessed through sparql queries. by enabling llms to automatically convert user questions into sparql, qa systems can provide accurate and rich responses by tapping directly into these vast repositories. furthermore, automating this process would reduce the need for defining the query manually, which will improve the accessibility to complex data for non-expert users. this line of research contributes to the development of knowledge graph question answering (kgqa) systems, which utilize kg to retrieve answers from structured data repositories. to assess the performance of these systems, a kgqa leaderboard has been established, as presented by the authors in @xcite , which allows the evaluation of kgqa systems using benchmark datasets. in this context, several benchmarking frameworks have been proposed, such as gerbil qa @xcite , which is designed to evaluate kgqa systems in a comprehensive way. among the widely used datasets for kgqa system evaluation, the question answering over linked data (qald) dataset series is considered a standard. in particular, the qald challenge foot_2 was launched to compare kgqa systems on various benchmarks, including qald-9plus @xcite and qald-10 (usbeck et al., 2023). furthermore, detailed evaluations tracking the progress of kgqa systems are available through the qald leaderboard foot_3 , providing valuable insights into the evolution of these systems. ## the analytic study in this analytic study, we fine-tuned several llms, all based on the llama architecture, including llama-3-8b foot_4 , llama-2-7b foot_5 , llama-3-70b foot_6 , and mixtral-8x7b foot_7 , to evaluate their performance in generating sparql queries from nlq and compare the results with existing similar systems reported in the kgqa leaderboard foot_8 and in @xcite . the reason to choose llama-based llms is both their availability, so they could be downloaded and fine-tuned locally, and their relative high performance in nlp related tasks. the models were trained and tested against two kgs, dbpedia and wikidata, providing a comprehensive comparison of their capabilities. the latest kgqa benchmark datasets for these kgs are provided in qald-9-plus foot_9 and qald-10 foot_10 respectively. the proposed method fine-tunes an llm such as meta-llama and mistralai-mixtral to transform nlq into sparql queries. the objective is to create a robust nlq-to-sparql transformation system capable of querying complex kgs accurately with minimal human input. since we, at this stage, only focus on questions in english, the first step involves filtering training datasets that pair nlqs with their corresponding sparql queries, retaining only english-language entries. these datasets, including benchmarks like 4 experiment all codes were written in python using the huggingface transformer library 13 . the models were trained and tested on two nvidia rtx a6000 gpus with 48 gb gddr6 memory. during the fine-tuning and the testing phases, we used dbpedia and wikidata qald datasets, which both include a training set and a testing set. dbpedia benchmark version is provided in qald-9-plus as a set of question-query pairs. the question is formulated in many languages, including english that we have used, and the query is the sparql translation of the question. qald-9-plus dbpedia training set contains 408 question-query pairs, and its testing set contains 150 pairs. wikidata is provided in qald-10 with a training set that contains 412 pairs and a testing set that contains 395 pairs. the experimental results, including detailed outputs for various models tested on qald-9-plus and qald-10 datasets, are available in the llama-kgqa github repository 14 . ## conclusion this study conducted an analysis that compared several llama-based llms for their ability to generate sparql queries from nlq. our results reveal that llama-kgqa, the fine-tuned version of llama-3-8b, has obtained a higher accuracy than larger models like llama-3-70b, while remaining efficient and scalable for real-world applications. the fine-tuning process using qald question-answering datasets has shown potential in enhancing the overall effectiveness and adaptability of our new qa model, llama-kgqa, marking a significant step forward in the application of llms within knowledge-driven ai. however, we also showed that for a kg (namely wikidata) which content would not have been transparent to the llm from its pretraining, especially due to non-human-readable identifiers, the performance llama-kgqa is dramatically lower. future work should therefore further explore incorporating the kg context into llm fine-tuning, which could improve the model's ability to interpret and generate more accurate queries. this perspective will target the challenges of using fine-tuned llms in efficient qa systems powered by knowledge graphs, in particular by enabling the llm to make use of information about relevant content in the knowledge graph during generation of the sparql query.