T 5 QL : Taming language models for SQL generation (2022)

Abstract: Automatic SQL generation has been an active research area, aiming at streamlining the access to databases by writing natural language with the given intent instead of writing SQL. Current SOTA methods for semantic parsing depend on LLMs to achieve high predictive accuracy on benchmark datasets. This reduces their applicability, since LLMs requires expensive GPUs. Furthermore, SOTA methods are ungrounded and thus not guaranteed to always generate valid SQL. Here we propose T5QL, a new SQL generation method that improves the performance in benchmark datasets when using smaller LMs, namely T5-Base, by 13pp when compared against SOTA methods. Additionally, T5QL is guaranteed to always output valid SQL using a context-free grammar to constrain SQL generation. Finally, we show that dividing semantic parsing in two tasks, candidate SQLs generation and candidate re-ranking, is a promising research avenue that can reduce the need for large LMs.

## introduction automated code generation has long been considered one of the fundamental tasks in computer science @xcite . recently, deep learning (dl) methods for code generation have been proposed which overcome the lack of flexibility of more traditional approaches @xcite . some dl approaches can act as code completion tools @xcite while others can use natural language (nl) as input to generate code @xcite , i.e., semantic parsing @xcite . the latter methods are particularly helpful for developers that are not proficient in all programming languages that are part of their development pipeline. for example, a developer might be familiar with the controller language (e.g., python) but unfamiliar with the database access language (e.g., sql). our method, t5ql, significantly improves upon t5-base and is superior to picard-base. picard-3b remains the sota for very large lms, i.e., picard-3b uses t5-3b which is ≈ 13x larger than t5-base. results for picard-base and picard-3b are straight (dashed) lines since @xcite only report results in the setting using database content for a single point, namely beam search with 4 beams. generating sql from nl is challenging because the nl query might be ambiguous (e.g., columns from different tables can have the same name). futhermore, obtaining labelled pairs of nl queries to sql is hard, time-consuming, and requires labellers that are proficient in sql. in recent years, benchmark datasets have been used by developers to evaluate their methods, namely spider @xcite and cosql @xcite . picard @xcite is the current sota method, i.e., the highest ranked method on spider. it is built on top of t5 @xcite , a general purpose llm. as proven by @xcite , llms are ungrounded and thus can generate any token at any given step, which may result in invalid sql; thus, to improve upon t5, picard prunes the search tree in order to avoid generating invalid sql. however, since picard fully prunes branches during beam search, it is not guaranteed to always generate an answer. another major issue with picard is that it needs a very large lm to achieve good performance: picard gets ≈ 75.5% exact match (em) accuracy in spider's development set when using t5-3b, but only ≈ 66.6% when using the smaller t5-base. here, we propose t5ql, a novel sql generation method that achieves 69.3% em on spider development set using t5-base instead of the ≈13x larger t5-3b. t5ql uses constrained decoding to ensure that it always generates valid sql, and it always generates an answer. our main contributions are: the remainder of the paper is organized as follows. section 2 presents sota for sql generation. section 3 describes t5ql's main components, namely constrained decoding and the ranker. section 4 shows our results. finally, section 5 concludes our work. ## state-of-the-art automated program generation has long been one of the major goals of computer science. various program synthesis tools have been proposed that generate sql from code fragments @xcite or pairs of input-output examples @xcite . however, code fragments might not be readily available if the developer does not write code or does not want to, and creating enough inputoutput examples for the program synthesis tool to be effective might be cumbersome. other tools generate sql from nl which is more developerfriendly @xcite . the complexity of generating sql from nl varies with the length and complexity of the sql query and the size of the database schema. thus, in order to properly evaluate and compare methods' performance, multiple benchmark datasets have been proposed, namely spider @xcite , spider-ssp @xcite , and cosql @xcite . we describe these benchmarks in detail in section 4.2 and discuss how they relate to our research questions (enumerated at the start of section 4). the current sota for sql generation (i.e., the methods that achieve the highest performance on benchmark datasets) comprises dl methods. dl methods for code generation avoid the complexity of traditional program synthesis and, thus, are generally faster during generation @xcite @xcite . ratsql's authors argue that predicting sql directly from nl is hard and can be made easier by instead predicting an intermediate representation (ir) that is more similar to nl than sql is @xcite . with this insight, they obtained sota results on spider. however, their ir is not capable of representing all sqls and, thus, for some queries the correct sql is not obtainable, leading to a loss of em accuracy. other approaches were built on top of ratsql with good results @xcite @xcite . one of the major disadvantages of these methods is that, since they use custom architectures, they cannot leverage pre-trained llms in their decoding step. being able to leverage llms is beneficial since they can be used for multiple tasks. for example, @xcite unifies structured knowledge grounding tasks into a text-to-text format and are thus able to train the same model for different tasks. to the best of our knowledge, @xcite were the first to propose a method that uses an llm, namely t5, and evaluate it on spider. they concluded that their method had good predictive capabilities, but sometimes generated syntactically incorrect sql and had lower precision in out-ofdistribution examples. since t5 is ungrounded, it cannot be guaranteed to always generate valid sql; the same is true for other llms @xcite . in order to address the issue, xiao et al. (2016) propose a method that constrains the output generation based on grammatical rules. they also compare a model trained with constraints and verify that using the constraints only during inference improves the model. targeting code generation specifically, @xcite propose picard, a method that constrains the model generation by removing wrong outputs during beam search. by doing so, picard is the current sota in the spider benchmark. however, they report that picard did not generate any sql for 2% of the queries. @xcite improve llm performance in the few-shot setting by introducing two components, one that selects the examples to be given to the model and another that constrains the generation of syntactically correct sql. however, fine-tuned models (e.g., picard) still perform better in the general task than their model, which was trained in the few-shot setting. in this work, we use one model to generate candidates, a generator, and another to re-rank them, a ranker. this choice is motivated by recent work @xcite where the authors show that a re-ranking method boosted performance for code generation. regarding semantic parsing more concretely, @xcite use a ranker model to select candidates, and then a fine-tuned model generates the final output; their model shows good generalization capabilities and outperforms previous methods for question answering on knowledge graphs. more recently, @xcite argue that when current llms are given a prefix prompt they can often generate text that is incoherent with the prefix. they propose a ranker model that scores the generator's candidates for an input prefix and obtain results that outperform earlier models in both automatic and human evaluation. ## method we start this section by presenting an overview of our method and its architecture (figure 2 ). then, we focus on each of its main components, namely constrained decoding and the ranker. finally, we discuss the scoring function and evaluation metrics. ## experiments we start by describing the experimental setup in section 4.1. then, we detail each dataset and the relevant evaluation metrics in section 4.2. then, each subsequent section (section 4.3-4.6) tries to answer each of the following research questions: q1. does constrained decoding improve the generator's performance? q2. does t5ql have compositional generalization capabilities? q3. does t5ql generalize to the conversational setting? q4. instead of using a very large generator, can we improve performance using a ranker? ## conclusion here we put forward t5ql, a new method for sql generation with sota performance on benchmark datasets when using small lms. t5ql uses constrained decoding to improve predictive performance and also to guarantee that the generated sql is always valid. futhermore, we complement the generator model with a ranker model that is capable of choosing the best candidate sql from a pool of a few candidates.