Beyond Read-Only: Crafting a Comprehensive C hinese Text-to- SQL Dataset for Database Manipulation and Query (2024)

Abstract: Text-to-SQL aims to convert natural language into structured query language, which is a challenging task. Current research focuses mainly on read operations and ignores other aspects of database operations such as create, update, and delete operations. The benchmark datasets as well as models that have been proposed also fail to cover these operations, limiting the development and practical applications in the field. To bridge this gap, we propose CRUDSQL, a large-scale cross-domain single-table CRUD operations Chinese Text-to-SQL dataset. The dataset contains 10,000 question/SQL pairs involving 625 tables from different domains. To support further research on this dataset, we also propose a baseline method, CRUDParser, which employs a two-phase approach based on BERT and T5 for SQL generation and incorporates two strategies, value matching, and value prompting, for interacting with databases to further improve the performance. The experimental results show that the new operation types bring different challenges for future research, and our approach achieves 67.08% and 83.8% exact set matching accuracy under both read and delete operations in the test set, but only 49.6% and 61.8% under create and update operations. We believe that the proposal of CRUDSQL as well as CRUDParser can provide new directions and possibilities for research and practical applications in the field of Text-to-SQL. The dataset is published at https://github.com/bizard-lab/CRUDSQL.

## introduction the text-to-sql task aims to transform natural language questions into corresponding sql query statements, and researchers have applied this technique to relational databases as natural language interface to database (nlidb) to help users interact with databases more easily. however, in the * corresponding author figure 1 : an example that introduces text-to-sql for crud operations. with the release of wikisql @xcite and spider @xcite , two large-scale textto-sql datasets, the field has gained more and more attention and development. wikisql is constructed based on wikipedia and contains 24,241 tables, 80,654 natural language questions, and corresponding sql queries. each of its databases has only one table, does not need to consider primary and foreign keys, and the form of sql to be predicted is relatively simple, and sead @xcite currently achieves the highest accuracy on this dataset, with 93%. spider contains 10,181 natural language question-sql pairs covering 138 different domains, which is a greater difficulty enhancement than wikisql which involves the correlation between multiple tables and the generation of more complex sql. currently, dail-sql @xcite achieves 86.6% accuracy on this dataset. however, current datasets and research efforts focus on read operations and ignore create, update, and delete operations. to fill this gap, this paper proposes crudsql, a large-scale cross-domain single-table crud operations chinese text-to-sql dataset. the dataset contains 10,000 questions and their corresponding sql queries involving 625 tables from different domains, in which the distribution of create, read, update, and delete is uniform. figure 1 shows an example of data from crudsql consisting of a table and questions oriented to the table about the crud operations and the corresponding sql statements. to the best of our knowledge, crudsql is the first text-to-sql dataset that contains all four operation types of create, read, update, and delete at the same time. in addition, we propose crudparser, a model that incorporates @xcite and t5 @xcite for two-stage sql generation as a baseline approach for this dataset. the first stage is sketch-based slot filling, where bert predicts the sql type corresponding to the question, as well as the sql column names, keywords, operators, and aggregation functions, and fills them into a predefined sql sketch. the sql sketch is then used as the basis for interaction with the database to improve the accuracy of the final sql generated. there are two types of interactions, one is value matching, which is used to improve the conditional value of the where clause in the sql statement, and the other is value prompting, which is used to optimize the values that are inserted into the database or the values that need to be updated. the second stage is the sequence-based generation, mainly used in update and create operations, where t5 can generate the final sql based on sql sketches as well as prompt words. the experimental results show that the create and update operations bring new challenges to the text-to-sql task, and the accuracy of these two operation types on the test set is only 49.6% and 65.6% even with the use of pre-trained language models as well as methods that have performed well in the past. although our value matching and value prompting strategies are effective, there is still a lot of room for improvement offered for further research. the key contributions of this paper are summarized as follows: ## benchmark approach in this paper, we propose a unified model called crudparser that aims to adapt create, read, update, and delete operations. the model combines two pre-trained language models, bert and t5, and employs both value matching and value prompting strategies. we divide the task as a whole into two phases: first, the prediction results of the bert model are utilized to populate the sketches; second, the t5 model is used to generate values involving update and create operations. ## conclusion in this paper, we propose crudsql, a large crossdomain, single-table chinese text-to-sql benchmark dataset with a special focus on four types of sql operations: create, read, update, and delete. compared to existing benchmark datasets that only have read operations, this paper introduces new challenges, especially for value linking, and puts more emphasis on the accuracy of the updated and created values. in addition to this, we propose a baseline approach for this dataset, crudparser, which generates sql in stages by pre-training the language model and incorporates value matching and value prompting strategies. we demonstrate the experimental benchmark results. this paper has demonstrated initial promise, but there is still room for improvement. one potential future work is to extend single-table to multi-table query sql generation although the focus of this paper is unifying manipulation and query sql producing. another future work is to fine-tune large language models on this dataset and put it closer to more practical application scenarios.