CSS : A Large-scale Cross-schema C hinese Text-to- SQL Medical Dataset (2023)

Abstract: The cross-domain text-to-SQL task aims to build a system that can parse user questions into SQL on complete unseen databases, and the single-domain text-to-SQL task evaluates the performance on identical databases. Both of these setups confront unavoidable difficulties in real-world applications. To this end, we introduce the cross-schema text-to-SQL task, where the databases of evaluation data are different from that in the training data but come from the same domain. Furthermore, we present CSS, a large-scale CrosS-Schema Chinese text-to-SQL dataset, to carry on corresponding studies. CSS originally consisted of 4,340 question/SQL pairs across 2 databases. In order to generalize models to different medical systems, we extend CSS and create 19 new databases along with 29,280 corresponding dataset examples. Moreover, CSS is also a large corpus for single-domain Chinese text-to-SQL studies. We present the data collection approach and a series of analyses of the data statistics. To show the potential and usefulness of CSS, benchmarking baselines have been conducted and reported. Our dataset is publicly available at https://huggingface.co/datasets/zhanghanchong/css .

## introduction given the database, the text-to-sql task @xcite aims to convert the natural language question into the corresponding sql to complete complicated querying. as the wild usage of relational database, this task attract great attention and has been widely studied in both academic and industrial communities. recently, text-to-sql researches @xcite @xcite mainly focus on building a parser under a cross-domain setup @xcite , where the databases of the training set and the evaluation set do not overlap. it aims to construct a universal parser that can automatically adapt different domains to inhibit the problem of data scarcity. however, domain-specific knowledge, especially domain convention, is crucial but difficult to transform across different domains under cross-domain setup. another line of research focuses on the experiment environment where the training data and the evaluation data are based on the same database, which is known as a single-domain setup. a single-domain text-to-sql system can parse domain knowledge more easily and also has more wide applications in the real world. however, the problem of data scarcity always comes up when security issues and privacy issues exist. therefore, both of these setups will face particular difficulties when it comes to the real world. to this end, we introduce the cross-schema setup in this work. the cross-schema text-to-sql tasks aim to build a text-to-sql parser that can automatically adapt different databases from the same domain, which can avoid the aforementioned problems. actually, the cross-schema text-to-sql also has broad applications in the real world. for example, all the hospital store the information of patients and medical resources in databases with different structures. most information categories are identical across these databases, for instance, the patient name and the treatment date. moreover, domain-specific representations such as medicine names in databases and user questions are also commonly used. in this case, we can build a universal in-domain text-to-sql parser that can be deployed on the new database from the given domain. compared with the cross-domain setup, a crossschema parser will not always confront completely unseen domain knowledge. on the other hand, compared with the single-domain setup, the problem of data scarcity can also be inhibited because the data from other in-domain databases can be used to train the model. however, a cross-schema text-to-sql parser need to automatically adapt different database schema structure. unfortunately, this issue is less investigated before. therefore, how to construct a structural-general parser is the mainly challenge of cross-domain text-to-sql. in this paper, we propose a large-scale cross-schema chinese text-to-sql dataset (css), containing 33,620 question/sql pairs across 21 databases. we generate (question, sql) pairs with templates and manually paraphrase the question by crowd-sourced. for the databases, we collect 2 real-world database schemas involving medical insurance and medical treatment. as the privacy issues, we are not allowed to use the original data. therefore, we fill the databases with pseudo values. based on these 2 seed databases, we alter the schema and expand 19 databases with different structures. hence, css can be used to develop cross-schema text-to-sql systems. on the other hand, the original 2 databases correspond 4,340 samples, which construct the largest chinese singledomain corpus. this corpus also allows researchers to carry on related studies. our main contributions can be summarized as follows: ## related works single-domain text-to-sql datasets earliest semantic parsing models are designed for singledomain systems to answer complex questions. atis @xcite ) contains manually annotated questions for the flight-booking task. geoquery @xcite ) contains manually annotated questions about us geography. @xcite ; @xcite ; @xcite convert geoquery into the sql version. restaurants @xcite is a dataset including questions about restaurants and their food types etc. scholar @xcite includes questions about academic publications and corresponding automatically generated sql queries. academic @xcite enumerates all query logics supported by the microsoft academic search (mas) website and writes corresponding question utterances. yelp and imdb @xcite consists of questions about the yelp website and the internet movie database. advising @xcite consists of questions about the course information database at the university of michigan along with artificial data records. single-domain text-to-sql datasets contain only one database. although text-to-sql models trained with single-domain datasets are applied in corresponding specific domains, different systems with the same domain but different backgrounds have diverse databases, which means that models should have the generalization ability to be transferred among different systems. existing singledomain datasets do not own the feature that requires models to improve cross-schema generalization ability. on the contrary, our cross-schema setup is raised for this issue. cross-domain text-to-sql datasets recent researches expect text-to-sql models @xcite @xcite to generalize to unseen databases. thus cross-domain text-to-sql datasets are released. @xcite releases wikisql, a dataset of 80,654 manually annotated question/sql pairs distributed across more than 20k tables from wikipedia. although wikisql is a large-scale dataset, each database schema merely consists of one table and each sql query merely consists of select, from, where clauses. @xcite releases spider, a large-scale complex cross-domain text-to-sql dataset. comparing with previous datasets, spider owns much more complex databases for various domains and complex sql queries with advanced sql clauses and nested sql structures. @xcite releases dusql, yet another large-scale cross-domain text-to-sql dataset but in chinese. having similar form with spider, dusql has become a popular chinese text-to-sql dataset. there are also some conversational cross-domain text-to-sql datasets, including sparc @xcite , cosql @xcite , chase @xcite , dir @xcite etc. although our cross-schema dataset owns more than one databases, it is different from crossdomain datasets. it concentrates on model generalization ability across different databases which share the similar structure since they are in the same domain. ## dataset collection in this section, we introduce our method of constructing the medical dataset css in detail. the dataset construction method mainly consists of five steps: 1) initial databases creation, 2) question/sql templates creation, 3) values filling, 4) questions rewriting, and 5) database schema extension. we discuss five steps of constructing the dataset in section 3.1-3.5 respectively. figure 1 shows the overview of the complete process. question template: 门诊就诊@xmath01 ? sql template: select temperature from outpatient_record where @xmath1: 门诊就诊123中患者的体温是多少? what is the temperature of the outpatient where visit id is 123 ? sql: select temperature from outpatient_record where @xmath2: 门诊就诊123中患者测了体温，数值是多少? what is the patient's temperature reading during the outpatient visit 123? sql: select temperature from outpatient_record where @xmath3 ## dataset statistics and comparison in this section, we list some statistical information of css and existing datasets and do comparison. we mainly discuss scale statistics and sql statistics with various datasets, including single-domain datasets, cross-domain datasets and css. ## conclusion this paper presents css, a large-scale crossschema chinese text-to-sql dataset designed for the medical domain. we illustrate the detailed process of dataset construction and also present statistical information comparing with existing datasets. we raise a challenge task in css, which requires models to generalize across various databases but in the same domain. to tackle the above task, we designed a baseline method named syntactic role prediction as an auxiliary task for model training. we conduct benchmark experiments with three competitive baseline models and prove that future researches on css is valuable.