S tat B ot. S wiss: Bilingual Open Data Exploration in Natural Language (2024)

Abstract: The potential for improvements brought by Large Language Models (LLMs) in Text-to-SQL systems is mostly assessed on monolingual English datasets. However, LLMs’ performance for other languages remains vastly unexplored. In this work, we release the StatBot.Swiss dataset, the first bilingual benchmark for evaluating Text-to-SQL systems based on real-world applications. The StatBot.Swiss dataset contains 455 natural language/SQL-pairs over 35 big databases with varying level of complexity for both English and German.We evaluate the performance of state-of-the-art LLMs such as GPT-3.5-Turbo and mixtral-8x7b-instruct for the Text-to-SQL translation task using an in-context learning approach. Our experimental analysis illustrates that current LLMs struggle to generalize well in generating SQL queries on our novel bilingual dataset.

## introduction switzerland is a multilingual country, officially recognizing four national languages: german, french, italian, and romansh. multilingualism is a pillar of the country's identity, ensuring that all citizens have equal access to public service, education, and any other information, regardless of their linguistic background. switzerland started offering open government data, and the supply continues to grow, spurred on, among other things, by open-by-default regulations of the federal government, and initiatives from cantons and municipalities. while the opendata.swiss initiative implements the once-only principle by offering a central catalog for all available swiss open government data, these datasets are often not standardized across administration levels and neither are methods of collection, compilation, and processing. if data are found, one must then work out and understand the † equal contribution. 1 we release our data and code to the community at https: //github.com/dscc-admin-ch/statbot.swiss methodological differences in order to know which data are more suitable for the intended usage and more importantly be capable of importing and analyzing the data through statistical software such as a spreadsheet, python, r or sas, which all require advanced computing skills. these challenges pose a risk for the democratic processes: the harder it is for a citizen to access an accurate source of information such as national statistics, the more likely a country is to suffer from misinformation. the statbot.swiss project aims to develop a swiss statistical bot that simplifies access to open government data by allowing interaction with the data directly via natural language. more precisely, one enters a question and gets an answer built upon the result of a query on trusted datasets from the opendata.swiss platform. the main objectives of this paper are (1) to provide a real-world bilingual dataset to benchmark text-to-sql systems, namely the statbot.swiss dataset, which consists of 455 intricate instances of querying data from 35 large databases, totaling 7.5 gb in size, available in both english and german, and (2) to assess the performance of current state-of-the-art pre-trained llms with various prompting strategies and so to establish a strong baseline on the statbot.swiss dataset. to the best of our knowledge, the statbot.swiss dataset is the first text-to-sql benchmark to incorporate english and german languages within the context of a complex text-to-sql benchmark and real-world databases. we conduct comprehensive evaluations on two pre-trained llms, namely gpt-3.5-turbo-16k, @xcite and @xcite . our experimental results show that current models using in-context learning strategies achieve up to 50.58% execution accuracy and struggle to generate sql queries giving exact matches on statbot.swiss, which shows that multilingual text-to-sql systems using state-of-the-art llms still lack robust-ness for reliable applications. multispider @xcite generalizes the spider benchmark to multiple languages. however, the drawbacks of multispider are also obvious. as a translated work from the spider dataset, the complexity of the database schema and the datasets are strongly limited, and the translation lacks native language expertise. unlike multispider, the statbot.swiss benchmark is realistic, complex, and curated by native speakers with proper domain knowledge, while covering both english and german for datasets with trusted sources. llms in text-to-sql translation recently, there has been significant development in employing large language models (llms) for the textto-sql task. several approaches have been suggested to improve the abilities of llms by incontext learning techniques @xcite @xcite or via semantic hypothesis re-ranking @xcite . additionally, methods such as intermediate reasoning steps and self-correction mechanisms have been integrated to enhance the performance of llms in text-to-sql applications @xcite . following these pioneering works, we combine in-context learning to llms for the text-to-sql task and conduct a comprehensive evaluation of prompt representations over the statbot.swiss benchmark. ## error analysis to delve deeper into the difficulties faced by stateof-the-art llms in the text-to-sql task using the statbot.swiss dataset, we have a closer look at the results of the best performing model, namely gpt-3.5, to perform an error analysis. we observe that (i) the model struggles significantly with inferring the queries involving group by multi-columns and mixed numeric operators. this leads to a low accuracy of 27.27% and 38.89% for these queries, respectively. these difficulties highlight the need for domain knowledge, which might not always be explicitly represented in the database schema, (ii) queries requiring proper built-in functions or other general sql keywords are answered slightly better by the llm, which counts for 40% and 44.44% accuracy, respectively, (iii) the language model results in an accuracy of 50% for null-values and 66.67% for nested select-alias, suggesting that the model has learned well from value illustration and grasped the use of sub-queries; see appendix d for examples, and (iv) finally, the outcome for with-queries and set-operation queries show an accuracy of 100%. however, the statistical significance of this performance is not guaranteed because of the small number of examples available in the dataset. for a more detailed understanding, we list nl/sql-pairs for each type of complex query in we particularly investigated datasets in which the underlying llm fails to answer any of their questions; see figure 6 in appendix f. in the tou rism_economy_by_canton english dataset, there exists a few false predictions due to the wrong understanding of data granularity. however, others are correctly labeled in terms of usefulness metrics, resulting in 66.67% of both ea soft and ea partial . given that each dataset was manually curated by various experts, we found that the observed disparity between ea strict and user's intent-based ea, namely soft and partial, is concentrated in certain datasets, which is in line with the previous assumption that semantic ambiguity is often subjective and individual. this insight of finding also demonstrates the importance of ea soft and ea partial from the experimental perspective. in the datasets of staatsausgaben_nach _aufgabenbereichen_cofog and gover nment_expenditure_by_function_cof og, the knowledge misunderstanding related to the granularity of the data led to the majority of wrongly generated queries. upon examining the similarity scores between the target question and the few-shot examples within each dataset, it becomes clear that there is a considerable level of disparity in the natural language questions found within these two datasets. ## conclusions our work highlights the potential and current limitations of large language models (llms) using our novel bilingual text-to-sql benchmark stat-bot.swiss. while llms like gpt-3.5-turbo and mixtral-8x7b show promising results for relatively simple queries, they face challenges with complex queries, domain knowledge inference, and handling null values. the analysis underscores the necessity for enhanced in-context learning in llms to incorporate a broader range of text-to-sql and database knowledge. the observed disparities in error analysis, especially in datasets with semantic ambiguities and data granularity issues, call for a more nuanced approach to evaluating the performance of text-to-sql systems, considering correctness from the user's perspective as apposed to exact matching. our work sets a benchmark for future developments in bilingual text-to-sql systems and emphasizes the importance of diverse and complex datasets in advancing llm capabilities. as the field progresses, refining these models to better understand and execute multi-column grouping, numeric operations, and domain-specific queries will be crucial for realizing their full potential in realworld applications. in future work, we aim to include other languages, e.g., french and italian, in our benchmark and extend towards cross-lingual text-to-sql tasks.