B ook SQL : A Large Scale Text-to- SQL Dataset for Accounting Domain (2024)

Abstract: Several large-scale datasets (e.g., WikiSQL, Spider) for developing natural language interfaces to databases have recently been proposed. These datasets cover a wide breadth of domains but fall short on some essential domains, such as finance and accounting. Given that accounting databases are used worldwide, particularly by non-technical people, there is an imminent need to develop models that could help extract information from accounting databases via natural language queries. In this resource paper, we aim to fill this gap by proposing a new large-scale Text-to-SQL dataset for the accounting and financial domain: BookSQL. The dataset consists of 100k natural language queries-SQL pairs, and accounting databases of 1 million records. We experiment with and analyze existing state-of-the-art models (including GPT-4) for the Text-to-SQL task on BookSQL. We find significant performance gaps, thus pointing towards developing more focused models for this domain.

## introduction relational databases are pervasive in all modernday organizations, from financial establishments to educational institutes. typically, query languages such as sql are used to extract the required data from relational databases. however, formulating queries in sql needs mastery of the language itself; consequently, this excludes people (particularly those without technical background, e.g., financial accountants) who do not know sql from using databases. it is imperative to develop techniques to address the research question, can relational databases be queried using natural language? in this paper, we take a step toward this goal; in particular, we explore if one could develop a natural language interface for accounting databases. in recent years, several large-scale general-purpose datasets @xcite have been proposed for ## related work due to its importance in practical applications, developing natural language interfaces to databases has been an active area of research. due to space constraints, we cannot cover all the research, and we refer the reader to the survey by @xcite . we outline some of the main works in this area in this section. several datasets have been proposed for text-to-sql task in recent years. for example, spider @xcite dataset has been proposed; it covers 138 different domains. a large-scale dataset, wikisql @xcite , consisting of 24241 wikipedia tables, has been created. similarly, squall @xcite , kag-gledbqa @xcite , and bird-sql @xcite datasets have been generated to evaluate the generalization property of models on unseen domains. domain-specific datasets have also been proposed, such as those based on yelp and imdb @xcite , advising domain @xcite , mimicsql @xcite , sede @xcite , booksql stats size of the database 1 million total businesses 27 size of question-sql pair 100k number of easy sql 10,000 number of medium sql 45,000 number of hard sql 45,000 restaurants domain @xcite , and academic domain @xcite . the purpose of these datasets is to evaluate the performance of models with a high degree of precision while disregarding the generalization characteristic of the models. comparison. we compare booksqlwith other popular datasets in ## booksql dataset given the importance and wide prevalence of business databases across the world, the proposed dataset, booksql focuses on the finance and accounting domain. accounting databases are used across a wide spectrum of industries like construction, healthcare, retail, educational services, insur-ance, restaurant, real estate, etc. business in these industries arranges their financial transactions into their own different set of categories (called a chart of accounts foot_2 in accounting terminology). for example, a restaurant business could have categories like advertising, license fees, etc., a real estate brokerage business could have categories like commissions, office supplies, etc. keeping generalization in mind booksql dataset includes a variety of businesses from different industries. hence, a text-to-sql system developed on booksql will be robust at handling various types of accounting databases. the total size of the dataset is 1 million. the dataset is prepared under financial experts' supervision, and the dataset's statistics are provided in ## baseline models we benchmark existing state-of-the-art (sota) text-to-sql models on booksql dataset. sede: we fine-tuned the sede model @xcite on the booksql dataset. sede is a t5-based sequence-to-sequence model @xcite . it takes unordered schema items (tables and column names) along with questions as input and generates the corresponding sql query as output. unisar: we fine-tuned the unisar model @xcite on the booksql train dataset, with t5-large as the base language model. unisar converts any seq-to-seq language model into a text2sql model by three non-invasive extensions nique as proposed in @xcite . it decomposes text-to-sql task into multiple subtasks and then solves each sub-task one by one by prompting gpt4 @xcite with sub-task-specific prompts. it uses the following sub-tasks: ## future directions results show the poor performance of the sota models on booksql. we outline some of the possible directions for the future to improve performance. multi-task learning: one could employ a multitask learning setup, i.e., in addition to optimizing for sql generation objective, adding other multitask objectives could help improve the performance on hard sql queries. these objectives could include (1) nested vs. non-nested sql classification, (2) distinct keyword classification, and (3) date format classification. pre-training: for large databases, it is difficult for any model to relate the question tokens with column names when the question might refer to some table cell value. before the text-to-sql task, ## conclusion in this paper, we propose booksql, a text-to-sql dataset that will have broad applications in the finance and accounting domain. the experimental outcomes of several text-to-sql models indicate considerable room for improvement. in the future, we aim to build a more robust model that can handle hard queries and improve performance.