Investigating Large Language Models for Text-to- SPARQL Generation (2025)

Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in code generation, such as translating natural language questions into SQL queries. However, state-of-the-art solutions often involve a costly fine-tuning step. In this study, we extensively evaluate In-Context Learning (ICL) solutions for text-to-SPARQL generation with different architectures and configurations, based on methods for retrieving relevant demonstrations for few-shot prompting and working with multiple generated hypotheses. In this way, we demonstrate that LLMs can formulate SPARQL queries achieving state-of-the-art results on several Knowledge Graph Question Answering (KGQA) benchmark datasets without fine-tuning.

## introduction the advent of large language models (llms) has significantly advanced the field of natural language processing (nlp), with particular success in the domain of code generation @xcite . at the same time, the growing complexity and scale of knowledge graphs @xcite @xcite highlighted the need for robust and accurate mechanisms to query such data stores, for instance, within knowledge graph question answering (kgqa) pipelines @xcite . in this paper, we carry out an extensive evaluation of llm-based in-context learning (icl) for text-to-sparql generation. text-to-sparql is a crucial component of many kgqa systems, that typically make use of different modules to assembly the query, such as entity and relation linking. to this end, we define a simple and modular approach to address the text-to-sparql task without fine-tuning. the evaluation is focussed on * work done while being at expert.ai. the following key aspects: (1) the influence of various in-context learning strategies on the quality of the generated queries; (2) the impact of different state-of-the-art model backbones, varying in architecture, size, and training data; (3) the potential of beam search to generate multiple query candidates, thereby enhancing the results;(4) a comparison between icl and specialized models finetuned for the task. the code is publicly available at https://github.com/jacopodabramo/dfsl . in the interest of reproducibility, as backbones, we use three state-of-the-art open-weight llms: mixtral 8x7b, llama-3 70b, and codellama 70b. we run experiments on two widely-used knowledge bases, dbpedia and wikidata, using four publicly available datasets: qald-9, based on dbpedia, and qald-9 plus, qald-10 and lc-quad 2.0, based on wikidata. our experimental results demonstrate that llms in-context learning solutions achieve state-of-theart results, without the need of any fine-tuning. the injection of demonstrations similar to the input question into the prompt combined with the generation of multiple query candidates directly from beam search hypotheses, yield the best results, exceeding in most of the benchmarks state-of-the-art models fine-tuned for the task. finally, we also run ablation studies to gauge the effectiveness of the approach without gold information from the el and rl modules. ## related work we first provide an overview of most related in-context-learning approaches. then, we discuss text-to-sparql methods, including kgqa systems that typically make use of text-to-sparql techniques to tackle the problem. ## method given a collection of natural language questions q and a knowledge graph g := (e, r, f), where e are entities, r are relations, and f ⊆ e × r × e are facts, kgqa is the problem of answering questions in q based on g. kgqa can be framed as a text-to-sparql task, where a question q must be translated into a sparql query s q to be executed on g by a sparql engine, to return a (possibly empty) answer a. the entities and relations in q, denoted as e q and r q , may be, and usually are, extracted from q before generating s q . hence, query generation can be tackled as a conditional text generation problem given q, e q and r q . within the scope of icl, p θ is a pre-trained llm and the conditional input e q , r q , q is combined with other contextual information c, such as additional instructions, guidelines, constraints and demonstrations, all expressed via natural language text. accordingly, the generated query is: ## experiments in this section, we conduct the investigation of llms capabilities in text-to-sparql query generation. kgqa serves as a benchmark task to measure the quality of the generated queries.