Addressing Limitations of Encoder-Decoder Based Approach to Text-to- SQL (2022)

Abstract: Most attempts on Text-to-SQL task using encoder-decoder approach show a big problem of dramatic decline in performance for new databases. For the popular Spider dataset, despite models achieving 70% accuracy on its development or test sets, the same models show a huge decline below 20% accuracy for unseen databases. The root causes for this problem are complex and they cannot be easily fixed by adding more manually created training. In this paper we address the problem and propose a solution that is a hybrid system using automated training-data augmentation technique. Our system consists of a rule-based and a deep learning components that interact to understand crucial information in a given query and produce correct SQL as a result. It achieves double-digit percentage improvement for databases that are not part of the Spider corpus.

## introduction text-to-sql is a task to automatically translate a given a natural language question into an sql formula for a specific database (db). in 2018, spider dataset @xcite was released as a large-scale complex and cross-domain semantic parsing and text-to-sql dataset. since then numerous models have been developed and evaluated using spider dataset. although there is a large increase in accuracy, growing from 20% to 70 + % between 2019 and 2021 of top ranked systems, later studies @xcite @xcite , show that the accuracy of these models drops significantly, in some cases well below 20%, for databases outside spider dataset. in this paper, we present various experiments for text-to-sql task using db schemas that are inside and outside of spider dataset. we identify systematic types of errors and analyze their possible causes. from our error analysis, we propose solutions to address the limitations of the underlying encoder-decoder architecture and of the spider dataset. we use training data augmentation technique in order to create a series of models for the same db schema. by coupling each model sql result with the information extracted from the query at the nlp step, we interpolate a final sql formula via ensemble technique. the system achieves very good results compared to its individual components and can be run on any new db schema, in or outside of the spider dataset, even when there exists a set of very limited number of training examples, on the order of several tens. this paper is organized as follows. in section 2, we review studies that defined a "standard" encoderdecoder architecture for text-to-sql. in section 3, we define the problem of encoder-decoder approach based on declined performances on new db schemas and new types of questions, that leads to the challenge of achieving good accuracy. next, we describe our system in section 5. the experiment results in section 6 and conclusions in section 7. ## related work the text-to-sql task is not a new task in nlp and there are many references on its complexity and achievements obtained recently @xcite @xcite . in @xcite , the spider corpus with thousands of training examples was introduced in 2018. the spider challenge has been introduced as well, where any team can have their system evaluated on an unknown corpus, which is a part of the spider corpus, but not publicly available. in @xcite @xcite , among others, a sequence-to-sequence approach was introduced, opening the way for this type of architecture. instead of translating into sql, @xcite @xcite , the system translates into a representation that captures the semantics of the query, called intermediate representation (ir). from ir to sql is a deterministic process: a context free grammar is used to convert one into another. the authors build on the work of @xcite that used abstract syntax tree (ast). a system that uses the the ir approach, but extends the context free grammar to include values, is valuenet(vl) @xcite . the code is available from @xcite . valuenet obtained an accuracy of 60% on spider development corpus, which put it among the best systems in the spider challenge in 2020. the rat system @xcite is based on the relation-aware self-attention mechanism, to address schema encoding, schema linking, and feature representation within a text-to-sql encoder. they augmented their system with bert @xcite , showing that using transformers brings a significant increase in the accuracy. but, most importantly, they also showed that "... all the known information about the schema, it is insufficient for appropriately encoding a previously unseen schema in the context of the question...". in @xcite , a detailed analysis of systems that performed well on spider corpus confirmed a large drop in accuracy for db schemas outside of the spider corpus. given that creating a new training is an expensive process, an important part of research for textto-sql task has been dedicated to improving the training methods, in order to reduce this cost. paraphrasing (ronak kaoshik, 2021), dialog @xcite , or a combination of these @xcite were used to cope with this problem. @xcite surveyed a series of methods to automatically create training corpora. some of these models are pretrained using the masked language modeling (mlm) task by either masking tokens from the utterance input or tokens from the schema input @xcite . this work demonstrated that jointly pre-training on utterances and table contents (e.g., column names and cell values) can benefit downstream tasks such as table parsing and semantic parsing. in @xcite a general method, which avoids the dependency of order in deep learning system, is presented. our own observation aligns very well with the observations in the previous papers, and confirms the difficulty to cope with new unseen db schema only by means of unique training corpus and/or model. however, our solution differs from what has been been proposed so far. the systems that used general pretraining methods, like rat+gap, did not show that the same performance is observed for out of spider schemas, but only for the ones in spider corpus. the systems that confronted the systematic errors that are produced in the case of new db schemas, took a different path from ours, like @xcite , and did not exploit systematically the relationship between values and filters as a means to control the correctness of sql formula. their line of research assumed that this state of fact is due only to the lack in training. however, our experiments support the idea, that, while indeed some errors are traceable to the particularities of a certain training corpus, one important cause of many errors is ignoring the way the db schema relationships could be expressed in natural language, see section 3. by explicitly providing the linkage between values and schema definition, coupled with specific training generation, and interpolating over an ensemble of different models, our system shows improvement for db schemas that are inside or outside of the same corpus. ## problem definition while the overall results for top performers on the spider corpus are high, the variation for individual db in the testing corpus can be large. however, when the same system is applied to the db schemas outside of the spider corpus we obtain a different distribution, figure 1 . the spider his-togram shows that accuracy for majority of test db schemas fell into the range of 40-90%. the data was obtained running the default vl, system trained on the spider training corpus and tested on the spider development corpus. the non spider general come from @xcite . none of these db schemas have an accuracy above 50%. the non spider custom refers to the training and test corpora for our own databases, described in section 4. for these databases, the accuracy is in the same category as for the lowest for non spider general, namely below 20%. this category contains just one db schema for the spider corpus. looking at what percentage of schemas are in three main categories, which roughly correspond to bad, ok, good, the difference between spider vs. non spider db schemas is very large. this shows that indeed there is a very strong skew in the distribution towards the left intervals of accuracy for new db schemas. the reason for this skew for new db schemas that is exemplified above was pointed out in @xcite and @xcite . without proper reference to the linkage between query tokens to specific table/columns, based only on the examples seen in training, the deep learning model is not able to discern between similar queries in the context of a new db schema. in section 4 we provide details on these types of errors and their causes. ## baseline and error analysis spider dataset @xcite is a large-scale cross-domain semantic parsing and text-to-sql annotated dataset. valuenet (vl) @xcite is an open source deep learning system built using an encoder-decoder architecture. the valuenet trained on spider dataset represents our baseline. our outside spider db schemas are hr, wh and bi (reference to articles kept anonymous for now). for each of these schemas we have from 70 to 200 training examples.these training examples are split into two different corpora ans -for training, and (2) nans -for test. vl obtained an accuracy of about 60% on the spider development, but much lower results for the new dbs. in table ## hybrid system (hs) we created a new system that processes and extracts the relevant information from the input question and links tokens to specific tables and columns from the corresponding db schema. we replaced valuenet's pre-processing and name entity recognition (ner) modules with our disambiguation dictionary module (ddm) to provide crucial information of the relationship between values and schema. we also introduced an ensemble of neural network models to improve the performance of the inferred sql queries. from the original valuenet architecture we kept only the encoder-decoder for ir with the semql grammar. the hybrid system has a different data flow from valuenet. first, an english query is processed by our natural language processing module (nlp). in our approach, we correlate technologies that increase the performance by double figure percentage: (i) we devise a seed training data augmentation technique (stda), that on the bases of the initial training corpus, we call seeds, is able to generate a larger training set. in a typical scenario, a few tens of seed examples lead to several hundreds of training examples; (ii) with different sets of automatically created training data, we train different models and use an ensemble technique to analyze the output of each of these models. ## experiments and results in this section we present the experiments and the results we obtained using the hybrid system against the baseline obtained by the original valuenet trained on the spider corpus. besides the three schemas presented in section 3, we also consider here the following schemas from the the spider corpus: pets_1 and dog_kennels. we considered specifically these two db schemas because the default system scored less than median on them. in in we also carried out a cross validation experiment on the 5 models obtained for each schema via stda. the results are in in 4.1 we analyzed certain patterns of errors. to see how much the hybrid system can improve on ea1, we compiled a specific corpus for multivalue question/queries examples for hr db schema. this corpus has 64 questions that have more than one value, such as what apple product has the largest stock in bestbuy? the the results we obtained applying the hybrid system only on the multiple values corpus are in the default vl is the the out-of-the-box vl system, default vl+tr is the same system trained on the extra training, hs , stda and ens stand for hybrid system, seed training data augmentation and ensemble techniques respectively. the ensemble algorithm works particularly well for this type of corpus. it is because the majority of wrong sql had the wrong reference to table and column, which makes them recoverable due to the ddm information used in ensemble. ea2 issue was related to the logically compound filters. because spider code cannot correctly process those types of sql elements, it is not fair to compare systems that employ any of our techniques vs. a system does not have any mechanism for dealing with compound logical operators. currently, we do not known of any other system that manages this problem. the efficiency of our solution is further proved by the experiments using temporal questions. the spider corpus have a poor representation of temporal questions, less than 8% in training and less than 7% in dev, and, as pointed out in ea3 the sql associated contains string , not date evaluation. we manually created a corpus containing temporal questions for hr and wh. this corpus contains questions with one data value, like, how many products were sold on 2/2/2014, we refer to them as tmp_1v, and questions that more than one data value, like how many products are sold with prices greater than 200 dollars between 2017/01/01 and 2017/12/31, we refer to as tmp_mv. for temp_1v the baseline system had a performance similar to the one for non temporal questions. however, for tmp_mv the situation change: the baseline system achieved 0% accuracy, and the situation did not improved when the available temporal training was provided to the system to build a new model. on the other hand, our system hs+stda+ens obtained a good accuracy, see ## conclusion and further research in this paper we present a hybrid system for the text-to-sql task. the architecture of this system was driven by seeking solutions to a problem that we found when we tried to run an encoder-decoder system with out-of-the-box configuration, namely a large drop in accuracy for our db schemas. the problem was reported in the results published by other teams as well. our solution was to construct a system that does schema linkage and employs a training data augmentation technique. the system showed a good accuracy for db schemas in and out of the spider corpus. we obtained a double figure improvement of the accuracy, in some cases close to 30%, and not less than 9%. the lowest increase was for a schema that was considered difficult in the spider corpus. our experiments strongly suggests that a perfect training corpus cannot be built such that the accuracy on any new schema will be high. however, by implementing schema linkage and specific training, automatically created, one can realize a high jump in accuracy for deep learning networks. our future research will focus on further improving the training with queries that are not properly represented in the spider corpus and on the methodology for training data augmentation.