Can LLM s substitute SQL ? Comparing Resource Utilization of Querying LLM s versus Traditional Relational Databases (2024)

Abstract: Large Language Models (LLMs) can automate or substitute different types of tasks in the software engineering process. This study evaluates the resource utilization and accuracy of LLM in interpreting and executing natural language queries against traditional SQL within relational database management systems. We empirically examine the resource utilization and accuracy of nine LLMs varying from 7 to 34 Billion parameters, including Llama2 7B, Llama2 13B, Mistral, Mixtral, Optimus-7B, SUS-chat-34B, platypus-yi-34b, NeuralHermes-2.5-Mistral-7B and Starling-LM-7B-alpha, using a small transaction dataset. Our findings indicate that using LLMs for database queries incurs significant energy overhead (even small and quantized models), making it an environmentally unfriendly approach. Therefore, we advise against replacing relational databases with LLMs due to their substantial resource utilization.

## introduction the advent of large language models (llms) has revolutionized several scientific and engineering disciplines, including software development tasks. many software development related tasks could be done or automatized by llms. the satisfactory performance of llm in search and query led to the introduction of specific llm databases such as vector database @xcite auxiliary knowledge information retrieval methods, a.k.a., retrieval augmented generation @xcite . relational databases are one of the oldest and most common components of software applications. these databases manage structured data using interconnected tables in tabular form. structured query language (sql) is the query language used to interact with relational databases. there are two widely known and significant limitations of using llms: (i) factual mistakes and hallucinations caused by neural networks @xcite , and (ii) token size limitations @xcite , which do not allow them to load a large dataset into their prompt, and thus have a limited data size. there are ongoing efforts to prove that the factuality and coverage of llms are quickly improving with new training architectures and the increasing amount of text used as input @xcite . besides, there are continuous efforts to increase or remove the token size limitation, such as using structured state space models (s4), e.g., mamba @xcite instead of transformers. our work does not quantify or tackle any of these two known challenges. it focuses on benchmarking resource utilization using llm instead of traditional sql. in this research, we intend to investigate whether llms could replace traditional database management systems to search and query tabular data. assuming even though the capability to generate sql queries exists in llms, we should measure resource consumption and how accurately it identifies the correct answers from tabular datasets. an essential consideration in our exploration is the environmental impact of llms. there are ongoing discussions 1234 on the huge electricity and water cooling supply, underscoring sustainability-related challenges brought about by the new existence and overall being of the llms. our results testify that even using a small-size trained llm still consumes a high amount of energy in comparison to a native sql engine running on a relational database. besides, we have observed the inferior accuracy of llms in comparison to sql engines. however, larger models might resolve the accuracy problem in the near future, but the energy issue remains open. ## literature review there are recent reports on the water and electricity consumption of generative artificial intelligence (ai) models, especially llms. however, their approach is mostly holistic and does not provide a comparative analysis of doing a particular task with llm and without llm @xcite @xcite . on the other hand, interest in adopting llms for general tasks like database querying has grown in the natural language processing community; there are several promising works in this direction, which we have categorized into two main groups. one group of work passes the query in natural language and data into an llm and, as a result, gets the sql query back. these works are known as text-to-sql @xcite @xcite @xcite . the latter group @xcite provides the data and the query in natural language as input into an llm. then, they get the result in natural language as well, we call them nlquery-to-nlanswer. in this section, we briefly describe each group of work. ## methodology in this work, we evaluate nine open-source llms that operate as nlquery-to-nlanswer. in particular, we measure their accuracy and resource utilization compared to sql queries. our study assesses how effectively llms are generating not only sql but also direct answers from natural language queries. as an sql engine, we choose to use (sqlite) foot_5 , which is a common sql engine used in devices that have resource constraints, such as android phones. there are promising tools available to measure the resource utilization of llms @xcite . however, we have used our scripts to have enough flexibility to measure different resources 7 . ## experimental evaluation we examine the resource usage of sql engine compared to llms to query tabular data, the proficiency of llms in generating sqlequivalent queries from natural language, and their effectiveness in obtaining semantically accurate responses from structured datasets. to establish a baseline for the evaluation of llms, we measure both the execution time and memory consumption for queries (a-j) associated with direct sql query execution. based on our measurement of the direct sql query execution on the sql engine, the average execution time is 0.41 ms, and the average memory usage is 1641 b. as we have described earlier, the sql engine we used is sqlite. the average execution time and memory utilization for direct query results and query generation of llm models are presented in tables 2 and 4 . moreover, we display the accuracy of direct query results by llm models in in the results shown in tables 3 and tables 5, symbols used are ✓ for correct generation and ✗ for incorrect or incomplete generation. ## discussion direct query results from llm models show disappointingly low accuracy. these findings highlight a significant challenge: llms struggle to query databases effectively without additional engineering. specifically in generating sql queries, models often misinterpreted complex requests, incorrectly applying sql clauses. our findings also point out that energy efficiency varies among llm models used for sql query generation, with larger models consuming more energy. quantized models, such as optimus-7b, performed well in the execution time and ## conclusion and future work llms offer a radically new perspective on database querying and the nature of computational systems. in this work, we measure the accuracy and resource utilization of nine small open-source llms in querying tabular data. our results present the significant resource expense of employing llms, even small models that are highly compressed with quantization. besides, the accuracy of using llm (at least not the very large and commercialized ones) for querying tabular data is low. as the model gets larger, the accuracy improves, but we did not experiment with larger models. potential further research can investigate fine-tuning existing models with sql schema, toward reducing the misinterpretations made by the llm models in querying databases.