Augmenting Multi-Turn Text-to- SQL Datasets with Self-Play (2022)

Abstract: The task of context-dependent text-to-SQL aims to convert multi-turn user utterances to formal SQL queries. This is a challenging task due to both the scarcity of training data from which to learn complex contextual dependencies and to generalize to unseen databases. In this paper we explore augmenting the training datasets using self-play, which leverages contextual information to synthesize new interactions to adapt the model to new databases. We first design a SQL-to-text model conditioned on a sampled goal query, which represents a user’s intent, that then converses with a text-to-SQL semantic parser to generate new interactions. We then filter the synthesized interactions and retrain the models with the augmented data. We find that self-play improves the accuracy of a strong baseline on SParC and CoSQL, two widely used cross-domain text-to-SQL datasets. Our analysis shows that self-play simulates various conversational thematic relations, enhances cross-domain generalization and improves beam-search.

## introduction multi-turn text-to-sql translation is a powerful semantic parsing paradigm that converts natural language user utterances into executable sql queries in a conversational environment. compared to regular text-to-sql tasks such as spider @xcite and geoquery @xcite , conversational text-to-sql requires interpreting coreference and omission phenomena that frequently appear in human conversations. to be effective, text-to-sql models must uncover complex contextual dependencies while grounding user utterances in task specific database schemas. numerous architectures and pretraining methods have been proposed for tackling context-dependent text-to-sql @xcite @xcite @xcite . however, the size of the datasets used has been limited due to the high cost of annotating multi-turn dialogue and sql pairs, which often requires trained experts. existing multi-turn text-to-sql datasets, such as sparc @xcite and cosql @xcite , require text-to-sql parsers to generalize to unseen databases at test time, but doing so is difficult with limited training context. in this paper we propose the use of self-play to augment multi-turn text-to-sql datasets in order to achieve more robust generalization. self-play simulates interactions between multiple artificial agents in order to generate a training signal in addition to supervised data. it has been successfully applied in a wide range of tasks, e.g. board games @xcite and multiplayer battle games @xcite . it has also been applied in dialogue simulations, during which a dialogue model converses with a user simulator to generate synthetic dialogues @xcite @xcite . in our work, we extend self-play to semantic parsing. although self-play has been adopted in taskoriented dialogue, the need to pre-define a domain specific ontology of slot-value pairs (e.g. the slot value "@xmath0. adding a new domain for task-oriented dialogue is difficult and labor-intensive. on the other hand, textto-sql tasks @xcite use a domain-independent formalism, i.e. sql queries. we demonstrate that self-play is well-suited to simulating interactions in a new domain given a database schema, improving cross-domain generalization. interaction select * from airlines select * from airlines where @xmath1: what are all the airlines? user: of these, which is jetblue airways? user: what is the country corresponding to it? airlines : id | airline | abbreviation ... airports : city | airport_code | airport_name ... ... flights : airline | flight_no | source_airport ... database gold interactions text-to-sql model sql-to-text model synthetic interactions filtered interactions train self-play retrain retrain sampled logical form filter utterances formal programs input: current utterance | database | previous utterances of these, which is jetblue airways? flight_2 | airlines : id, airline,... | airports: output: the next sql what are all the airlines? select * from airlines where @xmath2: the next utterance what is the country corresponding to it? input: user goal | previous utterances | last sql query | database select country from airlines where @xmath3, which is jetblue airways? what are all the airlines? select * from airlines where @xmath4: id, airline,... | airports: figure 1: multi-turn text-to-sql with self-play. we transform an interaction from sparc on the left to seq2seq formats (top: text-to-sql, bottom: sql-to-text). user utterances, sql queries, databases, and user goals are concatenated with a " | " symbol and shown in green, blue, yellow, and purple respectively. we use self-play to generate synthetic interactions. the synthetic interactions are filtered and used to retrain the text-to-sql and sql-to-text models. the text-to-sql model converses with the sql-totext model to simulate a new interaction. we filter out the interactions that are not grounded to the sampled goals and employ self-training @xcite to retrain the text-to-sql model and the sql-to-text model. we conduct extensive experiments on sparc and cosql. our main findings are: ## preliminary in this section, we formally define the multiturn text-to-sql task and introduce the picard @xcite model, which we use as our baseline. picard obtains state-of-the-art results on several text-to-sql tasks. ## method here we introduce how we use self-play for data augmentation. we first design a sql-to-text model ( §3.1). next, we describe how to use self-play to generate synthetic interactions ( §3.2). finally, we explain how we incorporate the generated data for self-training ( §3.3). ## datasets and main results in this section, we evaluate the performance of selfplay on cross-domain multi-turn semantic parsing. we first introduce the datasets ( §4.1), then detail the evaluation metrics ( §4.2), and finally we show the main results ( §4.3). ## analysis in this section we take sparc and systematically analyze the effect of self-play. first, to gain more insight into how a question's position or the query template affect the models, we examine self-play performance stratified by different turn number and sql templates in §5.1. then, we study whether self-play improves decoding during beam search ( §5.2). we further conduct a case study of self-play interactions in §5.3. ## conclusion we explore using self-play as a data augmentation method for generating synthetic dialogues in the cross-domain conversational semantic parsing task to address the challenge of data scarcity and cross-domain generalization. self-play learns various thematic relations in dialogues, improves beam search, and encourages the model's generalization to different domains. experiments on a t5 textto-sql semantic parser demonstrate the benefit of our proposed method. in the future, we will study using rewards in a rl setting to guide self-play to produce better synthetic dialogues.