M ulti SQL : A Schema-Integrated Context-Dependent T ext2 SQL Dataset with Diverse SQL Operations (2024)

Abstract: Text2SQL is a task that translates natural language into SQL statements. Context-dependent Text2SQL offers a more natural database interaction by simulating dialogues between users and databases, with CoSQL and SparC as representative datasets. Yet, these datasets struggle to accurately replicate real-world situations. To address this, we introduce MultiSQL, which extends them in three key aspects: (1) Diverse SQL Operations. We incorporate diverse SQL types such as Create, Update, and Insert to broaden the scope of SQL operations. (2) Schema-Integrated Context. We integrated query context with database schema dependencies to better depict database complexity. (3) Extended Dialogues. We expand dialogue length to better simulate long conversations and complex interactions. This multi-type, schema-integrated, context-dependent Text2SQL dataset comprises nearly 800 dialogue groups and over 9,000 interaction turns across 166 complex databases, offering a better benchmark for interactive user-database dialogue.Addressing MultiSQL’s challenges, we refined evaluation metrics to better capture diverse SQL types and schema dependencies. We designed a prompt framework that leverages historical data and self-refinement to accurately capture the dependency between text queries and database structures. Experiments with GPT-3.5, GPT-4, and LLaMA2-7B show both the effectiveness of our strategies and the challenges of MultiSQL. The datasets is available at https://github.com/grandchicken/MultiSQL.

## introduction in the information age, structured data is predominantly stored in databases. we interact with them through sql, a query language specifically designed for managing and manipulating databases. however, sql is quite complex, requiring a deep understanding of database structures and query syntax for effective data retrieval. to address this, the text2sql @xcite @xcite @xcite task was proposed, allowing direct conversion of natural language into sql queries. this enables users to interact with databases directly using natural language, greatly facilitating data access. to assess the effectiveness of text2sql, several datasets have been developed. spider @xcite is the first proposed dataset in a crossdatabase context. it includes complex sql types like groups, joins, and nested queries, effectively measuring model adaptability for this task. bird @xcite goes further towards real-world application, incorporating noisy data and external knowledge, to accurately depict the complexity of data in real-world business scenarios, thus challenging models in processing real-world data. however, the above datasets assume sql queries are stated in a single sentence, overlooking the reality that text2sql interactions typically occur as dialogues. in real-world scenarios, it's a more natural way for text2sql applications to involve users dynamically interacting with databases through dialogue. therefore, context-dependent text2sql has been developed, with sparc @xcite and cosql @xcite serving as pivotal datasets for this task. sparc simulates the user-database interactive dialogues, while cosql is a corpus developed for building cross-domain, general-purpose database querying dialogue systems. in the context-dependent text2sql task, models must understand the context to create the right sql statements for the current queries. however, existing context-dependent text2sql datasets still have limitations in accurately replicating real-world situations. to address this, we have extended them in the following three aspects: consequently, we propose a multi-type, schemaintegrated, and context-dependent text2sql dataset, called multisql covering various types of sql operations. it comprises nearly 800 dialogue groups, with over 9,000 turns spanning 166 complex databases. our dataset brings new challenges to method design and evaluation system construction. for methods, the integrated context necessitates that prediction processes thoroughly consider the semantic context generated during dialogue and changes in database structure. the presence of diverse sql types, along with longer dialogue histories, makes it easier for models to accumulate errors during interactions. to address this, we've developed a prompt framework, integrating historical data and a self-refinement @xcite mechanism to mitigate prediction errors. for evaluation, traditional metrics include exact set match and execution accuracy. however, for multisql, the former does not account for structural changes, and the latter struggles to handle issues with non-select statements that do not return values. consequently, we introduce context-aware match to track dialogue-induced table structure changes, and database state match to assess sql execution effectiveness by comparing changes of the database content. finally, our experiments on models like gpt-3.5 @xcite , gpt-4 @xcite , and llama2-7b @xcite highlight the difficulties inherent in the multisql dataset and the effectiveness of our proposed approaches. our contributions are summarized as follows: ## related work text-to-sql aims to automatically translate natural language questions into sql queries. the development of text-to-sql datasets has evolved to more closely resemble real-world scenarios, reflecting an increase in complexity and a move towards more accurately simulating real-world data interactions. initially, datasets in this field were relatively simple and focused on single-domain scenarios. early datasets such as geoquery (zelle and mooney, 1996), atis @xcite , and restaurant @xcite targeted specific information retrieval tasks within a limited domain. these datasets laid the groundwork for future advancements but were limited in their scope and complexity. considering that text2sql in real scenarios is an open-domain problem, the field has seen a shift towards cross-domain datasets, exemplified by wikisql @xcite and spider @xcite . these datasets broadened the scope from single-domain to cross-domain, requiring models to generalize across various domains. however, a common limitation in many cross-domain datasets is their focus on the database schema without adequately considering the specific values within the tables, diverging from the complexity of real-world databases. to address this, datasets like kaggledbqa @xcite , ehrsql @xcite , sede @xcite , and mimic-sql @xcite were introduced. these datasets focused on large-value databases and professional sql queries, thereby moving closer to real-world database applications. bird further advanced this trend by incorporating noisy data and external knowledge into its structure to accurately depict the complexity of data in real-world business scenarios, making it more challenging and representative of real-world data processing scenarios. another significant development in text-to-sql datasets is the recognition that it's more natural for text2sql applications to involve users dynamically interacting with databases through di-alogue. therefore, context-dependent text2sql has been developed. sparc simulates user-database interactive dialogues, requiring models to understand and respond to context-dependent queries. cosql, on the other hand, is designed for building cross-domain, general-purpose database querying dialogue systems. it challenges systems in sqlgrounded dialogue state tracking, response generation, and user dialogue act prediction, closely mimicking real-world dialogue scenarios. ## dataset construction due to cosql being a representative example of a context-dependent text2sql dataset, we adopted its database in our work. building upon this foundation, we constructed a text2sql interaction dataset to further explore and address the complexities of natural language interfaces to databases. due to the high cost and extensive time requirements associated with manual data annotations, we use large language models as an aid in the dataset construction process like much recent research does @xcite @xcite . we designed a set of prompt frameworks and used gpt-4 to generate data. subsequently, we manually corrected and modified the generated data, and iteratively checked and improved the quality of the dataset. below, we will provide a detailed description of the entire dataset construction process. ## data statistics and analysis figure 3 shows the distribution of sql types in our dataset, excluding delete statements which are rare in practice. the balance among other operations is notable, with insert at 25.83%, select at 25.8%, and update at 23.19%. this reflects the varied use of sql in real situations, enhancing our dataset's relevance and providing a broad testing ground for systems handling diverse sql queries within dialogues. ## method multisql, with diverse sql operations, schema-integrated context, and extended dialogues, presents unique challenges to the method design. first, the challenge arises from the need to understand the semantic context within dialogues and adapt to database schema changes. to address this, we integrate historical data, a strategy that involves incorporating past interactions into the model's current decision-making process. this approach ensures that the model not only grasps the immediate query but also contextualizes it within the dialogue's history, enhancing its ability to adapt to and reflect changes in the database structure ac-curately. by doing so, we mitigate issues related to semantic understanding and database adaptation, ensuring more accurate and contextually relevant predictions. second, the diversity in sql types and the potential for error accumulation through extended dialogues require a robust mechanism to maintain model accuracy. here, our solution is the implementation of a self-refinement mechanism. this process involves continuously analyzing model predictions for errors and refining the model's strategies based on feedback. such a mechanism directly tackles the accumulation of errors by enabling the model to learn from its mistakes and adjust its approach for future queries, thus enhancing its reliability and accuracy in handling a wide range of sql operations within prolonged dialogues. thus, we have devised a prompt framework that integrates historical data and incorporates a selfrefinement mechanism. the framework contains several parts as shown in figure 4: • system initialization: starts with a prompt that outlines the task, database schema, and initial data context. in this way, our approach systematically addresses the challenges posed by multisql through initial setup, context integration, and iterative refinement. section 7 experiments further validate the effectiveness of our methodology. database llm prompt framework user @xmath0: this metric assesses the equivalence of the predicted query to the gold query across specific sql components: from, where, group by, etc. each component is evaluated for an exact match between the predicted and gold queries. the predicted query is deemed correct if, and only if, all components match exactly with their counterparts in the gold query. however, these metrics encounter specific challenges when applied to our dataset. ## conclusion in conclusion, we introduce multisql, a multitype, schema-integrated, and context-dependent text2sql dataset, designed to closely mirror the complexities and dynamism of real-world database interactions. by incorporating a diverse range of sql operations and embedding database schema dependencies within extended dialogue interactions, multisql offers a significantly more nuanced and challenging environment for text2sql applications.