Battle of the Large Language Models: Dolly vs LL a MA vs Vicuna vs Guanaco vs Bard vs C hat GPT - A Text-to- SQL Parsing Comparison (2023)

Abstract: The success of ChatGPT has ignited an AI race, with researchers striving to develop new large language models (LLMs) that can match or surpass the language understanding and generation abilities of commercial ones. In recent times, a number of models have emerged, claiming performance near that of GPT-3.5 or GPT-4 through various instruction-tuning methods. As practitioners of Text-to-SQL parsing, we are grateful for their valuable contributions to open-source research. However, it is important to approach these claims with a sense of scrutiny and ascertain the actual effectiveness of these models. Therefore, we pit six popular large language models against each other, systematically evaluating their Text-to-SQL parsing capability on nine benchmark datasets with five different prompting strategies, covering both zero-shot and few-shot scenarios. Regrettably, the open-sourced models fell significantly short of the performance achieved by closed-source models like GPT-3.5, highlighting the need for further work to bridge the performance gap between these models.

## introduction text-to-sql parsing automatically converts user input questions into sql statements, enabling the retrieval of relevant information from databases. by enabling users to express their objectives in natural language, text-to-sql systems could minimize the technical obstacles for non-expert users to interact with relational databases and enhance productivity. the introduction of large pre-trained language models like bert @xcite and t5 @xcite has further improved the performance of text-to-sql systems. researchers have been leveraging the impressive understanding * work done when donovan was working at i 2 r of these models to push the boundaries of text-to-sql capabilities. recently, breakthroughs in decoder-based large language models @xcite have further revolutionized the field of nlp. a prominent trend is the pursuit of training increasingly larger language models, encompassing billions of parameters, and utilizing vast amounts of textual data. subsequently, these models are fine-tuned using instruction-based techniques, enabling them to better adhere to human-generated text prompts. among the prominent applications of decoder llms is chatgpt, which is built upon openai's gpt-3.5 and gpt-4 models. chatgpt has demonstrated exceptional capabilities in zero-shot and few-shot scenarios, as evidenced by various textto-sql evaluation studies @xcite . regrettably, the success of chat-gpt has ignited an ai race, leading industry research labs to cease public disclosure of their model parameters and training methodologies. therefore, researchers have been actively pursuing the development of new language models that can potentially rival the capabilities of chat-gpt. these models include dolly, which builds upon the pythia models @xcite , as well as vicuna @xcite and guanaco @xcite , based on the llama models @xcite . some of them have garnered attention by claiming to achieve performance levels surpassing 90% of gpt-4 through fine-tuning techniques. as practitioners of text-to-sql, we are grateful for the contributions made by these models. however, we remain uncertain about whether these open-source models truly deliver the level of quality they claim to achieve. to address this concern, this paper presents a comprehensive evaluation of six language models: dolly, llama, vicuna, guanaco, bard, and chatgpt, directly comparing their performance on nine benchmark datasets, utilizing five distinct prompting strategies. our main findings are: to facilitate further research in the field of text-to-sql parsing, we are making all raw and post-processed outputs from the large language models publicly at https://github.com/ zhangyuchenyc/deepeval-text2sql . ## related work recently, decoder-based large language models have contributed tremendously to the codegeneration tasks @xcite @xcite . these models leverage unsupervised auto-regressive learning on large-scale text data, allowing them to capture rich semantic relationships and probability distributions of words. despite their remarkable performance with just one or few-shot examples in context, recent research suggests that they still face challenges on the textto-sql task, which involves complex reasoning @xcite . there are several works which focus on improving the text-to-sql parsing capabilities of large language models through enhanced prompt designs. in one study conducted by @xcite , the authors emphasize the significance of carefully selecting examples for in-context learning. they demonstrate that incorporating syntactic structures from example queries can greatly enhance the few-shot capabilities of large language models. chang and fosler-lussier (2023) conducted a comprehensive study that explores the impact of prompt length on the performance of text-to-sql models. additionally, they examine the sensitivities of representations of database knowledge across various domains. @xcite propose a case-based reasoning framework that adjusts the inputs for gpt-3.5 in cross-domain settings by adaptively retrieving case prompts. @xcite improve the generalization capabilities of large language models with boundary-based techniques that preprocess prompts at both token-level and sequencelevel of schema and sql. concurrently, some studies have also explored the potential benefits of complex, multi-step reasoning in improving the performance of large language models on text-to-sql parsing. @xcite show that least-to-most prompting @xcite might be unnecessary and directly applying chain-of-thought (cot) prompts @xcite could lead to error propagation. @xcite introduce a divide and prompt paradigm for the text-to-sql task, which involves dividing the task into multiple subtasks and applying the cot approach to each subtask. in another study by @xcite , a self-correction module is employed in a zero-shot setting to achieve a new state-of-the-art result on the spider leaderboard. this module feeds the solution of each sub-problem back to the large language model, enabling it to construct a better overall solution. ## conclusions and future work this paper systematically evaluates the text-to-sql parsing capabilities of six popular large language models across nine benchmark datasets, using five distinct prompt strategies. our findings indicate that the open-sourced models fall significantly short in performance when compared to closed-source models. however, it is worth noting that even gpt-3.5 performs worse than smaller baseline models on several classical datasets. we are making our outputs available for further analysis and to facilitate future research endeavors. there are several research topics we want to explore in the future. firstly, we plan to investigate the fine-tuning of these large language models on text-to-sql datasets using limited gpu resources using techniques such as low-rank adaptation @xcite . second, we want to explore methods that can dynamically select examples for in-context learning. third, we would also explore applying model compression techniques to recent text-to-sql models @xcite . lastly, we are interested in examining the feasibility and limitations of employing these large language models on multi-turn text-to-sql datasets, such as the sparc @xcite .