Chase: A Large-Scale and Pragmatic C hinese Dataset for Cross-Database Context-Dependent Text-to- SQL (2021)

Abstract: The cross-database context-dependent Text-to-SQL (XDTS) problem has attracted considerable attention in recent years due to its wide range of potential applications. However, we identify two biases in existing datasets for XDTS: (1) a high proportion of context-independent questions and (2) a high proportion of easy SQL queries. These biases conceal the major challenges in XDTS to some extent. In this work, we present Chase, a large-scale and pragmatic Chinese dataset for XDTS. It consists of 5,459 coherent question sequences (17,940 questions with their SQL queries annotated) over 280 databases, in which only 35% of questions are context-independent, and 28% of SQL queries are easy. We experiment on Chase with three state-of-the-art XDTS approaches. The best approach only achieves an exact match accuracy of 40% over all questions and 16% over all question sequences, indicating that Chase highlights the challenging problems of XDTS. We believe that XDTS can provide fertile soil for addressing the problems.

## introduction the problem of mapping a natural language utterance into an executable sql query in the crossdatabase and context-dependent setting has attracted considerable attention due to its wide range of applications @xcite . this problem is notoriously challenging, due to the complex contextual dependencies among questions in a sequence. consider the question sequence in figure 1 . in order to understand the last question, one needs to figure out the elliptical object of the verb "培养(have)" from the first two questions in the sequence, which is "状 元 球员(first pick player)". questions like this are context-dependent, since they require resolutions of contextual dependencies such as ellipsis in this question. there are also context-independent questions that can be understood individually, such as the first question in figure 1 . for ease of reference, we refer to this cross-database context-dependent text-to-sql problem as xdts. to study the challenges in xdts, a continuous effort has been dedicated to constructing datasets, including sparc @xcite and cosql @xcite . however, through a careful analysis on existing datasets, we identify two biases in them and these biases conceal the major challenges in xdts to some extent. first, there are only a limited number of context-dependent questions in existing datasets. specifically, only 32% of questions in cosql are context-dependent, and only 66% of question sequences have context-dependent questions. sparc has more context-dependent ques-tions than cosql, but it still has 48% of contextindependent questions. such a limited number of context-dependent questions is unexpected, because prior work @xcite has shown that questions within a database dialogue are highly likely to be context-dependent, and how to effectively model the context to understand a context-dependent question is one of the major challenges in xdts. second, 40% of sql queries in both sparc and cosql are particularly easy, involving at most one condition expression. this biased distribution of sql queries is potentially caused by their construction methods. in fact, we find that sql queries for question sequences created from scratch are much more challenging. upon identifying the limitations of existing datasets, we present chase, a large-scale and pragmatic chinese dataset for xdts. chase consists of 5,459 question sequences (17,940 questions with their sql queries annotated) over 280 multitable relational databases. compared with sparc and cosql, the number of context-independent questions in chase is reduced from 48% and 68% to 35%, and the number of easy sql queries is reduced from 40% and 41% to 28%. moreover, chase has richer semantic annotations, including the contextual dependency and schema linking @xcite of each question. chase is also the first chinese dataset for xdts. chase is made up of two parts: chase-c and chase-t. in chase-c, we recruit 12 chinese college students who are proficient in sql to create question sequences from scratch and annotate corresponding sql queries. to ensure the diversity and cohesion of question sequences, we propose an intent recommendation method. when a student is going to raise a question, an intent category is randomly sampled with the method, and the student is recommended to write the question and sql query according to it. in chase-t, inspired by the construction of cspider @xcite , we translate all the questions, sql queries, and databases in sparc from english to chinese. we also try our best to mitigate the biases in sparc. to understand the characteristics of chase, we conduct a detailed data analysis and experiment with three state-of-the-art (sota) xdts approaches, namely, editsql @xcite , igsql @xcite , and our extension of rat-sql @xcite . the best approach only achieves an exact match accuracy of 40% over all questions and 16% over all question sequences, indicating that chase presents significant challenges for future research. the dataset, benchmark approaches, and our annotation tools are available at https://xjtu-intsoft.github.io/chase . in summary, this paper makes the following main contributions: ## study of existing datasets in this section, we first formally define the problem of xdts and its evaluation metrics. then, we present our study to understand the limitations and biases of existing datasets in contextual dependency and sql hardness distribution. ## dataset construction given the limitations of existing datasets, we present chase, a large-scale and pragmatic chinese dataset for xdts. unlike the construction of sparc and cosql, we do not specify a final goal for each question sequence. instead, we motivate our annotators to raise diverse and coherent questions via an intent recommendation method. based on this method, we collect a set of relational databases, and we recruit annotators to create question sequences from scratch and annotate corresponding sql queries. data collected in this way are referred as chase-c. besides, inspired by the construction of cspider @xcite and vietnamese spider @xcite , we translate all the questions, sql queries, and databases in sparc from english to chinese. during translation, we also try out best to mitigate the biases in sparc. data collected with this method are referred as chase-t. chase is make up of both chase-c and chase-t. since all existing datasets for xdts are constructed for english, prior work on this problem primarily focuses on english, leaving other languages underexplored. to enrich the language diversity, in this paper, we construct chase for chinese, and we leave the support of more languages as our important future work. ## data statistics and analysis we compute the statistics of chase and conduct a thorough analysis to understand its three characteristics: contextual dependency, sql hardness distribution, and mention of database schema items. ## experiments to understand the performance of the sota approaches on chase, chase-c, and chase-t, we experiment with the three approaches introduced in section 2.2. appendix a.3 provides the details of our adaptations for chinese inputs and the experimental setup. ## conclusion and future work this work presents chase, to date the largest dataset for xdts, consisting of 5,459 question sequences over 280 databases. each question in chase has rich semantic annotations, including its sql query, contextual dependency, and schema linking. experimental results show that chase highlights the challenging problems of xdts and there is a long way for us to achieve real textto-sql demands of users. currently, chase is constructed for chinese. we plan to support more languages in the future. besides, we plan to explore the ways to utilize the rich semantic annotations in chase to address the challenges in xdts.