K aggle DBQA : Realistic Evaluation of Text-to- SQL Parsers (2021)

Abstract: The goal of database question answering is to enable natural language querying of real-life relational databases in diverse application domains. Recently, large-scale datasets such as Spider and WikiSQL facilitated novel modeling techniques for text-to-SQL parsing, improving zero-shot generalization to unseen databases. In this work, we examine the challenges that still prevent these techniques from practical deployment. First, we present KaggleDBQA, a new cross-domain evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. Second, we re-examine the choice of evaluation tasks for text-to-SQL parsers as applied in real-life settings. Finally, we augment our in-domain evaluation task with database documentation, a naturally occurring source of implicit domain knowledge. We show that KaggleDBQA presents a challenge to state-of-the-art zero-shot parsers but a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%, doubling their performance.

## introduction text-to-sql parsing is a form of database question answering (dbqa) that answers a user's natural-language (nl) question by converting it into a sql query over a given relational database. it can facilitate nl-based interfaces for arbitrary enduser applications, thereby removing the need for domain-specific ux or learning query languages. as such, dbqa attracted significant attention in academia and industry, with development of supervised datasets @xcite , large-scale models @xcite , and novel modeling techniques @xcite . the key challenge of text-to-sql parsing is zeroshot generalization to unseen domains, i.e. to new database schemas and differently distributed nl questions. large-scale annotated datasets like spider @xcite and wikisql @xcite evaluate cross-domain generalization of textto-sql parsers by restricting overlap between train and test domains. such challenging benchmarks facilitate rapid progress in dbqa. state-of-the-art (sota) accuracy on spider rose from 12.4% to 70.5% in just two years since its release, demonstrating the value of well-chosen evaluation settings. despite impressive progress in dbqa, deployment of sota parsers is still challenging. they often lack robustness necessary to deploy on reallife application domains. while many challenges underlie the gap between sota dbqa and its reallife deployment, we identify three specific discrepancies. first, spider and wikisql datasets normalize and preprocess database schemas or rely on academic example databases that originate with humanreadable schemas @xcite . in contrast, industrial databases feature abbreviated and obscure naming of table, columns, and data values, often accrued from legacy development or migrations. figure 1 shows a characteristic example. after deployment, text-to-sql parsers struggle with schema linking to domain-specific entities because they do not match the distribution seen in their pre-training (e.g. bert) or supervised training (e.g. spider). second, the nl questions of spider and wik-isql have high column mention percentage @xcite , which makes their language unrealistic. this can be an artifact of rule-generated nl templates (as in wikisql) or annotation uis that prime the annotators toward the schema (as in spider). either way, real-world deployment of a text-to-sql parser optimized on spider faces a distribution shift in nl, which reduces its realistic performance. finally, the standard evaluation setting of crossdomain text-to-sql parsing assumes no in-domain database: student math score table finrev_fed_17: state_code school_district yr_data t_fed_rev c14 c15 ⋮ 33 new york city school district 17 2061297 956851 439209 ⋮ 47 fairfax co schs 17 126916 21035 36886 ⋮ column descriptions: t_fed_rev total federal revenue through the state to each school district c14 federal revenue through the state-title 1 (no child left behind act) c15 federal revenue through the state -child nutrition a table finrev_fed_17_key: state_code state #_records 1 alabama 137 ⋯ ⋯ ⋯ 50 wisconsin 425 51 wyoming 48 example question: which school district received the most of federal revenue through state in wisconsin? example sql: select t1.school_district from finrev_fed_17 as t1 join finrev_fed_key_17 as t2 on t1.@xmath0.state_code where t2.@xmath1.t_fed_rev desc limit 1 figure 1: two table excerpts from the student math score database in kaggledbqa and an example question-sql pair. the column names are abbreviated (e.g. t_fed_rev) or obscure (e.g. c14, c25) but documentation (e.g. column descriptions) alleviates this. source: https://kaggle.com/loganhenslee/studentmathscores . supervision. this simplifies parser evaluation and raises the challenge level for zero-shot generalization. however, it does not leverage knowledge sources commonly present in real-world applications, both explicit (annotated in-domain examples) and implicit (e.g. database documentation, sql queries in the application codebase, or data distributions). a well-chosen alternative evaluation setting would facilitate development of dbqa technologies that match their real-world evaluation. ## kaggledbqa: a real world dataset the goal of the kaggledbqa evaluation dataset is to more closely reflect the data and questions a text-to-sql parser might encounter in a real-world setting. as such, it expands upon contemporary cross-domain text-to-sql datasets in three key aspects: (i) its databases are pulled from real-world data sources and not normalized; (ii) its questions are authored in environments that mimic natural question answering; (iii) its evaluation assumes the type of system augmentation and tuning that could be expected from domain experts that execute text-to-sql parser deployment. we describe each of these components in turn in this section. ## conclusion & future work kaggledbqa provides two resources to facilitate real-world applications of text-to-sql parsing. first, it encourages an evaluation regime that bridges the gap between academic and industrial settings, leveraging in-domain knowledge and more realistic database distribution. we encourage adopting this regime for established text-to-sql benchmarks. second, it is a new dataset of more realistic databases and questions, present-ing a challenge to state-of-the-art parsers. despite the addition of domain knowledge in the form of database documentation, our baselines reach only 26.77% accuracy, struggling to generalize to harder questions. we hope that better use of documentation and new modeling and domain adaptation techniques will help further advance state of the art. the kaggledbqa dataset is available at https://aka.ms/kaggledbqa .