Re-appraising the Schema Linking for Text-to- SQL (2023)

Abstract: Most text-to-SQL models, even though based on the same grammar decoder, generate the SQL structure first and then fill in the SQL slots with the correct schema items. This second step depends on schema linking: aligning the entity references in the question with the schema columns or tables. This is generally approached via Exact Match based Schema Linking (EMSL) within a neural network-based schema linking module. EMSL has become standard in text-to-SQL: many state-of-the-art models employ EMSL, with performance dropping significantly when the EMSL component is removed. In this work, however, we show that EMSL reduces robustness, rendering models vulnerable to synonym substitution and typos. Instead of relying on EMSL to make up for deficiencies in question-schema encoding, we show that using a pre-trained language model as an encoder can improve performance without using EMSL, giving a more robust model. We also study the design choice of the schema linking module, finding that a suitable design benefits performance and interoperability. Finally, based on the above study of schema linking, we introduce the grammar linking to help model align grammar references in the question with the SQL keywords.

## introduction recent years have seen great progress on the textto-sql problem, i.e. translating a natural language (nl) question into a sql query @xcite @xcite @xcite @xcite , with neural networks the de facto approach. to achieve good performance on text-to-sql tasks, a neural model needs to correlate natural language queries with the given database schema, a process called schema linking. previous work often explicitly designs a module to perform the schema linking, which we term exact match based schema linking (emsl) @xcite @xcite . specifically: although emsl has been widely used and helps models obtain the state-of-the-art performance on some text-to-sql benchmarks @xcite , in this work, we show that emsl renders models vulnerable to noise in the input, particularly synonym substitution and typos. we then investigate whether text-to-sql models can preserve good prediction performance without emsl. previous ablation studies @xcite claiming the necessity of the schema linking module were conducted without pretrained language models (plms) such as bert. in fact, we find that when a pretrained language model is used as a model encoder, removing emsl has very little impact on the performance of the model. this observation is consistent for different model architectures and training schemes, such as ratsql @xcite , gnn @xcite , and gap @xcite . based on this finding, we introduce a more reasonable text-to-sql encoder design. we evaluate the models in three settings: the original spider benchmark without input noise @xcite , the spider-syn version with synonym substitution @xcite , and a new typo injection setting. results show that the use of a plm can provide the same performance benefit as emsl, while achieving better robustness against synonym substitution and typos. removing emsl also allows the model to obtain better results when training with synonym substitution samples. we also show that mas (multi-annotation selection, @xcite , a method designed to improve model robustness with emsl, can also improve models without emsl. in conclusion, we demonstrate emsl is no longer a necessary building block of text-to-sql models. the emsl and pretrained language models are part of the schema linking module that learns a score to decide which schema item to select. there are two design choices to compute this score: the first sees it as a direct relation between the question and schema items @xcite ; while the second considers the question and schema items together @xcite . we show that our best model employing the first design choice achieves the state-of-the-art performance, while its schema linking scores are also more interpretable than models with the second design. despite the shortcomings of emsl, it still significantly improves previous models: without a clear idea of how to align the entity references in the question and the schema columns or tables, emsl plays an important role in alignment. inspired by emsl, we propose instead exact match grammar linking (gl), which improves the model's ability to generate correct sql keywords. in short, this work investigates the role of the schema linking module for text-to-sql models, regarding its effect on model performance and robustness. we summarize our key findings below: ## proposal i: construct the schema linking without the emsl our first proposal is that the schema linking should not include emsl. schema linking itself is essential for text-to-sql models, but emsl can be replaced with a better mechanism. ## proposal ii: schema linking module design choice we believe that a text-to-sql model with good performance can ignore the schema linking feature, but it must include a schema linking module. while implementation details of such models differ, figure 3 : two text-to-sql schema linking design choices. the common factor is the calculation of a similarity score between each question word and schema item: correct schema items should obtain higher similarity scores. schema linking modules output attention scores from computing the schema linking feature and word embeddings. there are currently two attention mechanism designs. the first calculates a score that relates the question on one side, to the schema items on the other. the second approach also considers the attention scores between different schema items @xcite , thus it takes the question and schema items together as input to produce the attention scores. in figure 3 , we elucidate the inputs and outputs associated with the two types of attention computations. what is the average horsepower of the cars before 1980 ? # # # agg col # # table grsm year # select avg(horsepower) from cars_data where @xmath0: what is the name of stadiums with highest average attendance # # # col # table in jjs col # select name from stadium order by average desc limit 1 example 2: question with its grammar linking labels and sql first design is more interpretable: it requires the model to infer the correct schema item from the question, so you can see which part of the question is related to each selected schema item. with the second design, it is sometimes difficult to explain why the specific schema item is chosen due to the presence of some other schema items. for ease of understanding, we show and discuss the attention and interpretability of an example under different design choices in appendix b; we test the effects on performance in section 5.4. ## proposal iii: exact match grammar linking (gl) the mechanisms of existing models limit their performance in some specific situations discussed in section 4.2 and appendix c. our third proposal is gl which can provide ideas for addressing these limitations. ## experiments 5.1 generating typos to evaluate robustness against typos, we randomly insert a letter into the correct schema annotation word. (this is enough to break emsl, so we do not also modify the question words). we generated three typo development sets, named spider-t1 to spider-t3. the typos in spider-t1 are generated by randomly inserting a letter at any position except the end. in contrast, spider-t2 appends a random letter at the end of the schema annotation words. we examine these separately: the bert tokenizer may be able to split spider-t2 typos into a correct word and a suffix, but is less likely to split the spider-t1 typos well. we convert every schema annotation word in spider-t1 and t2 to typos when word length is greater than five letters; typos are generally more likely to occur in longer words, and words with more than five letters account for about 40% of the dataset. spider-t3 is then the same as spider-t1, but only converts the most frequent schema item words to typos. while spider-t1 and t2 simulate the impact of large numbers of typos in extreme cases, spider-t3 evaluates the impact of a more realistic, smaller number of typos. other typos are possible, e.g. by deleting and swapping letters; we discuss these in appendix d. ## conclusion in this work, we investigate the role of schema linking for text-to-sql models regarding model performance and robustness. in particular, we demonstrate that by leveraging pretrained language models, emsl is no longer a necessary building block to ensure a high performance on text-to-sql benchmarks. we observe that when emsl is used, models become overly reliant on it, making them vulnerable to attacks that break the exact-match assumptions of emsl. beyond this, by studying different schema linking module designs we find that a direct relation between the question and schema items is more interpretable and works well with intermediate representation sql. finally, inspired by emsl, we introduce exact match grammar linking for dealing with some examples where existing models can easily make mistakes.