STAR : SQL Guided Pre-Training for Context-dependent Text-to- SQL Parsing (2022)

Abstract: In this paper, we propose a novel SQL guided pre-training framework STAR for context-dependent text-to-SQL parsing, which leverages contextual information to enrich natural language (NL) utterance and table schema representations for text-to-SQL conversations. Concretely, we propose two novel pre-training objectives which respectively explore the context-dependent interactions of NL utterances and SQL queries within each text-to-SQL conversation: (i) schema state tracking (SST) objective that tracks and explores the schema states of context-dependent SQL queries in the form of schema-states by predicting and updating the value of each schema slot during interaction; (ii) utterance dependency tracking (UDT) objective that employs weighted contrastive learning to pull together two semantically similar NL utterances and push away the representations of semantically dissimilar NL utterances within each conversation. In addition, we construct a high-quality large-scale context-dependent text-to-SQL conversation corpus to pre-train STAR. Extensive experiments show that STAR achieves new state-of-the-art performance on two downstream benchmarks (SParC and CoSQL), significantly outperforming previous pre-training methods and ranking first on the leaderboard. We believe the release of the constructed corpus, codebase and pre-trained STAR checkpoints would push forward the research in this area.

## introduction text-to-sql parsing @xcite @xcite aims to translate natural language (nl) questions into executable sql queries, which enables the users here, each database schema refers to the table/column names of databases and each schema state refers to a slot-value pair, whose slot is a column/table name (e.g., degrees.campus) and its value is a sql keyword (e.g., select). "x" indicates that the semantic/intent is switched between turn2 and turn3 utterances. despite the remarkable progress of previous talms, they still suffer from technical challenges in the context-dependent setting. first, existing talms merely explore contextual information to enrich utterance representations without considering the interaction states determined by history sql queries, which are relevant to the user intent of current utterance. nevertheless, the trace and usage of historical sql information can contribute greatly to model the current sql query, as sql conveys user intent in a compact and precise manner. as shown in figure 1 , the second sql query is more likely to select the contents from the "compuses" table since the first sql query mentioned that table. although tracking schema states is essential to keep track of user requests for context-dependent text-to-sql parsing, how to model, track and utilize schema states throughout a conversation has not yet been explored in previous talms. second, context-dependent textto-sql parsing needs to effectively process context information so as to help the system better parse current nl utterance, since users may omit previously mentioned entities as well as constraints and introduce substitutions to what has already been stated. taking figure 1 as an example, the second utterance omit the implicit constraint of "campuses in year 2000" as mentioned in the first utterance. however, most prior talms primarily model stand-alone nl utterances without considering the context-dependent interactions, which result in suboptimal performance. although score @xcite model the turn contextual switch by predicting the context switch label between two consecutive user utterances, it ignores the complex interactions of context utterances and cannot track the dependence between distant utterances. for instance, in figure 1 , score fails to capture the long term dependency between the first and the fourth utterances since there is a switch between the second and the third utterances. in this paper, we propose a novel pre-training framework star for context-dependent text-to-sql parsing, which explores the multi-turn interactions of nl utterances and sql queries within each conversation, respectively. first, we propose a schema state tracking (sst) objective to keep track of sql queries in the form of schema-states, which predicts the value (a sql keyword) of each schema slot of the current sql query given the schema-state representation of previously predicted sql query. by introducing the schema-states to represent sql queries, we can better capture the alignment between the the historical and current sql queries, especially for the long and complex sql queries. second, we propose an utterance dependency tracking (udt) objective to capture com-plex semantic dependency of sequential nl questions, which employs weighted contrastive learning to pull together semantically similar nl utterances and push away dissimilar nl utterances within each conversation. a key insight is that the utterance corresponding to similar sql will be more semantically relevant, as sql is a highly structured indication of user intent. concretely, we propose two novel similarity functions (sql semantic similarity and sql structure similarity) to comprehensively construct appropriate positive and negative nl question pairs. we summarize our main contributions as follows. (1) to the best of our knowledge, we are the first to propose a schema state tracking (sst) objective for context-dependent talm, which tracks and updates the schema states of the context-dependent sql queries in the form of schema states. (2) we propose an utterance dependency tracking (udt) objective to capture complex semantic information of sequential nl questions, which employs weighted contrastive learning with two novel sql-oriented similarity functions to pull together two semantically similar nl utterances and push away the representations of dissimilar nl utterances within each conversation. (3) we construct a high-quality large-scale context-dependent text-to-sql conversation corpus to pre-train star. experiments show that star achieves new state-of-the-art performance on two downstream benchmarks (sparc and cosql) and ranking first on the leaderboard. ## task definition in this section, we first provide the formal task definition for context-dependent text-to-sql parsing. let @xmath0, . . . , u t } denote the utterances in a context-dependent text-to-sql conversation with t turns, where u i represents the i-th nl question. each nl sentence u i contains n i tokens, denoted as u @xmath1, . . . , w n i ]. in addition, there is a corresponding database schema s, which consists of n tables {t i } n @xmath2. the number of columns of all tables in the schema is m. we use s i to denote the name of the i-th item in schema s. at current turn t, the goal of text-to-sql parsing is to generate the sql query o t given the current utterance u t , historical utterances {u 1 , . . . , u t-1 }, schema s, and the last predicted sql query o t-1 . star primarily consists of a stack of transformer layer, which converts a sequence of l input tokens @xmath3, ..., x l ] into a sequence of contextualized vector represen- ## pre-training objectives as illustrated in figure 2 , we propose two novel pre-training objectives sst (schema state tracking) and udt (utterance dependency tracking) to explore the complex context interactions of nl utterances and sql queries within each text-to-sql conversation, respectively. in addition, we also employ the mlm (masked language modeling) objective to help learn better contextual representations of the conversations. next, we will introduce the pre-training objectives in detail. ## data construction for pre-training the cost of expensive sql annotation poses a challenge to the construction of large scale pretraining data. previous work @xcite resort to data augmentation to address this issue. typically in a conversational setting, contextdependent data augmentation techniques require two steps: (1) single-turn context-free grammar for utterance-sql pair generation, and (2) a follow-up context-free grammar to expand single-turn data into context-dependent conversations. score synthesized a total of 435k text-to-sql conversations following this setup, and we noticed two limitations with it. firstly, it relies on the template-filling construction to convert sql to utterances, resulting in rather rigid generated utterances in step (1). secondly, sparc is the only data resource employed to induce the follow-up context-free grammar in step (2). nevertheless, the contextual diversity in sparc is insufficient to simulate complex contextual dependencies. to this end, we propose a new pre-training data construction method. inspired by the snowball framework @xcite , we harness a generative model, i.e., bart, to bring more diversity to the generated utterances. for the follow-up conversational context-free grammar induction, we consider both cosql and sparc datasets and man ## conclusion in this paper, we proposed star, a pre-trained talm, which could jointly learn user utterance and table schema representations for context-dependent text-to-sql conversations. star contained two novel pre-training objectives (schema state tracking and utterance dependency tracking) to explore the complex context interactions of nl utterances and sql queries within each text-to-sql conversation, respectively. we constructed a diverse large-scale context-dependent text-to-sql conversation corpus to pre-train star. experiments demonstrated that star achieves new state-of-the-art performance on sparc and cosql.