PG - GSQL : Pointer-Generator Network with Guide Decoding for Cross-Domain Context-Dependent Text-to- SQL Generation (2020)

Abstract: Text-to-SQL is a task of translating utterances to SQL queries, and most existing neural approaches of text-to-SQL focus on the cross-domain context-independent generation task. We pay close attention to the cross-domain context-dependent text-to-SQL generation task, which requires a model to depend on the interaction history and current utterance to generate SQL query. In this paper, we present an encoder-decoder model called PG-GSQL based on the interaction-level encoder and with two effective innovations in decoder to solve cross-domain context-dependent text-to-SQL task. 1) To effectively capture historical information of SQL query and reuse the previous SQL query tokens, we use a hybrid pointer-generator network as decoder to copy tokens from the previous SQL query via pointer, the generator part is utilized to generate new tokens. 2) We propose a guide component to limit the prediction space of vocabulary for avoiding table-column dependency and foreign key dependency errors during decoding phase. In addition, we design a column-table linking mechanism to improve the prediction accuracy of tables. On the challenging cross-domain context-dependent text-to-SQL benchmark SParC, PG-GSQL achieves 34.0% question matching accuracy and 19.0% interaction matching accuracy on the dev set. With BERT augmentation, PG-GSQL obtains 53.1% question matching accuracy and 34.7% interaction matching accuracy on the dev set, outperforms the previous state-of-the-art model by 5.9% question matching accuracy and 5.2% interaction matching accuracy. Our code is publicly available.

## introduction text-to-sql is a sub-task of semantic parsing, which aims to convert utterances to sql queries. a great deal of deep learning approaches @xcite @xcite @xcite have been proposed to solve context-independent text-to-sql tasks such as wikisql @xcite and spider @xcite . among them, sqlova @xcite achieves 84.2% and 83.6% logical form accuracy on wikisql dev and test sets, and rat-sql @xcite achieves 69.7% and 65.6% exact matching accuracy on the spider dev and test sets when accessing database content, respectively. however, in real-world interaction scenarios, users often interact with the system through multi-turn dialogue. then, the system needs to generate complete sql query based on historical interaction information and current user utterance, as shown in figure 1 . @xcite propose a seq2seq model with an interaction-level encoder to solve domain-specific context-dependent task called atis @xcite which only contains the flight-booking domain. unlike atis dataset, @xcite present sparc, a new complex cross-domain and context-dependent text-to-sql dataset based on spider. most of previous approaches in the field of text-to-sql only focus on translating stand-alone utterances to sql queries, and are not suitable for solving sparc which contains contextual dependencies. database: tvshow table : tv_channel, tv_series, cartoon utterance 1: tell me the package option for the series named "rock tv". utterance 2: tell me the language of this series. utterance 3: list the language used least number of tv channel. list language and number of tv channel. sql query1: select package_option from tv_channel where @xmath0: select language from tv_channel where @xmath1: select language , count(*) from tv_channel group by language order by count(*) asc limit 1 figure 1 : an example of interaction from sparc dataset @xcite . the generation of each sql query depends on the historical information and current utterance, except for the first sql query. cd-seq2seq @xcite ) is a baseline model of sparc and it do not consider the table-column dependency, which causes the errors in predicting columns when tables are mentioned in the question but columns show ambiguity. recently, @xcite propose editsql which contains an editing mechanism that treats the previous sql query as a sequence and reuses the previous sql query to achieve the new state-of-the-art performance on sparc. but editsql generates table and column together (e.g., tv channel.series name) in a token during decoding phase, which raises noise of table prediction. moreover, we observe that foreign key dependency is significant in cross-domain text-to-sql datasets @xcite @xcite which contain more than 95% sql queries that depend on foreign keys to link tables in both train and dev sets. last but not least, to our knowledge, there is no research on utilizing tables to guide the prediction of sql queries in decoding phase. it is worth studying whether this method will improve the performance. in this work, we propose a novel model called pg-gsql to address the cross-domain contextdependent text-to-sql task. our encoder is based on interaction-level encoder, and we use pointergenerator network (pg) with guide (g) component as the decoder. the interaction-level encoder and pointer-generator network are employed to capture the historical information in encoding and decoding phases, respectively. the pointer part in pointer-generator network is used to copy a token from previous sql query and the generator is used to generate a new token from vocabulary in each decoding step. the guide component is designed to avoid dependency errors which contain table-column dependency and foreign key dependency errors during token generation phase. in addition, we observe that linking table and its corresponding columns improves the prediction accuracy of tables, hence we design a novel column-table linking mechanism to concatenate table and its corresponding columns together to represent the table. experimental results show that pg-gsql achieves 34.0% question matching accuracy and 19.0% interaction matching accuracy on sparc dev set. when using bert augmentation, pg-gsql obtains 53.1% question matching accuracy and 34.7% interaction matching accuracy on sparc dev set, gains 5.9% question matching accuracy and 5.2% interaction matching accuracy improvements compared to the previous state-of-the-art model. ## related work the task of translating utterance to sql query has a long history @xcite @xcite @xcite , and the earlier works focus on specific database and add extra manual intervention to generalize each database @xcite @xcite . recently, with the advent of large-scale cross-domain text-to-sql datasets @xcite @xcite , methods based on neural network are applied to the context-independent text-to-sql tasks such as wikisql and spider @xcite @xcite @xcite @xcite . however, the work most relevant to our research is the cross-domain context-dependent text-to-sql task. sparc @xcite and cosql @xcite are the latest datasets of cross-domain context-dependent text-to-sql, and cosql is more complex than sparc because cosql includes system responses. there is not enough research in cross-domain context-dependent text-to-sql task, which requires a model to depend on the historical information and current utterance to generate sql query from the unseen database. @xcite propose a seq2seq model with an interaction-level encoder to solve atis @xcite , and this model is extended to sparc called cd-seq2seq @xcite . cd-seq2seq uses position embeddings to get the position information for each utterance, and proposes a copy mechanism to copy segments from the previous sql query. in addition, the interaction-level encoder uses a discourse state to maintain and update over entire interaction. @xcite propose syntaxsql-con based on syntaxsql @xcite , the difference between them is that syntaxsql-con uses two bi-directional lstms @xcite with different parameters to encode the previous utterance and current utterance. both of them utilize the sql syntax and generation history to generate sql queries. recently, @xcite propose editsql to employ the interaction-level encoder with the utterance-table encoder as the encoder. moreover, editsql proposes a table-aware decoder with an editing mechanism as the decoder. the editing mechanism is similar to our hybrid pointer-generator network, which is used to copy tokens from the previous sql query or insert new tokens. our hybrid pointer-generator network is inspired by @xcite which focuses on abstractive text summarization. the difference between our model and the previous model in pointer-generator network is that we copy words from previous sql query rather than the source sentences. ## pg-gsql in this section, we present the model pg-gsql to tackle the cross-domain context-dependent text-to-sql task. similar to cd-seq2seq @xcite , we employ an interaction-level encoder as pg-gsql encoder. the decoder consists of two effective innovations. (1) we use the pointer-generator network as decoder to copy tokens from previous sql query or generate new tokens from vocabulary. ## experiment in this section, we evaluate the effectiveness of our model on two large complex cross-domain contextdependent text-to-sql datasets: sparc contains 3034, 421, 842 interactions and cosql contains 2164, 292, 551 interactions for training, development and testing. as the test sets of sparc and cosql are unreleased, we only evaluate our model on their dev sets. we evaluate our model using question matching accuracy (the exact set matching score over all questions) and interaction matching accuracy (the exact set matching score over all interactions). we do not use any extra data to boost our model for fair comparison. in addition, we conduct an ablation study to analyze the contribution of each innovation on sparc dev set. ## conclusion in this paper, we present pg-gsql with hybrid pointer-generator network and guide component to address the cross-domain and context-dependent text-to-sql task. experimental results show pointergenerator network is capable of reusing the previous sql query tokens to significantly improve interaction matching accuracy. furthermore, guide component avoids table-column dependency and foreign key dependency errors during decoding phase, and column-table linking improves the prediction accuracy of tables. the ablation study shows that our model improves the performance not only in predicting simple queries, but also in predicting nested, complex queries in unseen databases.