Towards Knowledge-Intensive Text-to- SQL Semantic Parsing with Formulaic Knowledge (2022)

Abstract: In this paper, we study the problem of knowledge-intensive text-to-SQL, in which domain knowledge is necessary to parse expert questions into SQL queries over domain-specific tables. We formalize this scenario by building a new benchmark KnowSQL consisting of domain-specific questions covering various domains. We then address this problem by representing formulaic knowledge rather than by annotating additional data examples. More concretely, we construct a formulaic knowledge bank as a domain knowledge base and propose a framework (ReGrouP) to leverage this formulaic knowledge during parsing. Experiments using ReGrouP demonstrate a significant 28.2% improvement overall on KnowSQL.

## introduction text-to-sql translates user queries into executable sql, greatly facilitating interactions between users and relational databases. along with the release of large-scale benchmarks @xcite @xcite and developments in model design @xcite ), textto-sql works are now achieving promising results in both research and practical applications @xcite . however, in the professional application of textto-sql, such as in the data analysis of financial reports, models require external knowledge to map the expert query with the domain-specific database. take the financial query for example: what's the ebit 1 of walmart?, where the underlying database has component columns that can be used to calculate the ebit. we treat this problem as knowledgeintensive text-to-sql, where domain knowledge is highly necessary to parse expert questions over domain-specific tables. this problem prevents textto-sql techniques from being fielded in novel, professional applications to assist the experts in processing data. traditional approaches would address this problem by annotating specific question/sql pairs on a target domain @xcite . then such mappings are induced during the training process. this approach does work but has the drawback that any induced information is both fragile and expertise-heavy: such knowledge does not port across domains and requires expert knowledge to craft. we propose to solve this problem by modeling how a non-expert person might tackle this problem. when meeting unseen examples (as in the ebit case above), they may first search for the related mathematical formulas from public resources, then ground the concepts referenced in the formulas with schema elements presented in their particular databases. this process leverages common, encoded formulaic knowledge that are already described in publicly-available resources such as tu-torials, textbooks, encyclopedias, and references. inspired by this, we propose to address the knowledge-intensive text-to-sql through formulaic knowledge which provides the evidence of mapping from domain-specific phrases presented in questions to actual sql operations over schema elements. more concretely, we define a taxonomy of three types of formulaic knowledge: calculation, union, and condition, each corresponding to a particular snippet of sql. then we propose re-group, a text-to-sql framework (fig. 1 ), consisting of three stages: (1) retrieve the formulaic knowledge from formulaic knowledge bank as an external knowledge source; (2) ground the concept of formulaic knowledge to the schema elements (e.g., exports to ship_out); (3) parse the results with the question, schema, and grounded formulaic knowledge. the external formulaic knowledge bank imbues regroup with formulaic knowledge, making it knowledgeable. regroup is also extensible because updating the formulaic knowledge bank does not require retraining any modules. moreover, we construct a chinese benchmark knowsql, to examine the effectiveness of re-group framework. it advances the existing knowledge-intensive text-to-sql beyond the previous work @xcite by considering more sql operations and challenging domains. experimental results demonstrate the re-group with formulaic knowledge would improve the performance by 23.4% overall. furthermore, we classify error cases into three classes, which are resolvable by advancing the corresponding module of regroup. finally, we discuss the potential future work such as expanding the scope of knowledge and advancing regroup model design. our contributions are summarised as follows: ## regroup framework to address the knowledge-intensive text-to-sql problem, we propose a novel framework named regroup, consisting of three stages: (1) retrieve the formulaic knowledge from the formulaic knowledge bank as an external knowledge source; (2) ground the concept of formulaic knowledge to the schema elements (e.g., exports to ship_out); ## experimental results and analysis to evaluate our approach: regroup with external formulaic knowledge bank, we conduct several experiments on knowsql benchmark. we report both the overall results of the pipeline and the finegrained results of each module. we also conduct error analysis and categorize the bad cases into three main classes. note that we report the average experimental results of each setting during three runs foot_6 . ## discussion is formulaic knowledge better than textual knowledge for text-to-sql? in sec.3.1, we argue that formulaic knowledge is preferred over textual knowledge intuitively. empirically, we conduct the experiments by the following steps: (1) transforming the formulaic knowledge to textual knowledge through annotators; (2) training the retriever and parser with textual knowledge under the same experiment setting as formulaic knowledge. experimental results reveal that textual knowledge receives an overall performance degradation of 13.6% compared with what's the cost of collecting formulaic knowledge? during the collection process of formulaic knowledge bank (19 domains), we found most domains have the public knowledge resource. moreover, the effort spent on collection formulaic knowledge is also acceptable compared with annotating data examples. for example, we spent 4 hours in collecting 219 formulaic knowledge in the finance domain, which is far more effective than annotating the equivalent data examples. eventually, formulaic knowledge improves the performance by 35.0% without retraining the model as shown in how to expand the scope of formulaic knowledge further? in this paper, we mainly focus on domain knowledge and mathematical knowledge and transfer them into formulaic knowledge format for model learning. other types of knowledge would improve the knowledge-intensive textto-sql further, such as the commonsense knowledge (e.g., water freezing point: @xmath0.g., favourite food: tiramisu). thus, we could package these types of knowledge into a formulaic format in future work. ## conclusion and future work this paper explores formulaic knowledge to address the knowledge-intensive text-to-sql problem, which would advance the professional application of text-to-sql such as data analysis for domain experts. first, we analyze the challenge of knowledge-intensive text-to-sql and construct a new challenging benchmark knowsql. then we propose to address this problem from the view of formulaic knowledge. concretely, we propose a simple framework regroup to leverage an external formulaic knowledge bank. experimental results reveal that regroup with formulaic knowledge achieves the 28.2% improvements overall. we further discuss three directions in improving the regroup via analyzing different types of bad cases: (1) iterative filling in the blank of formulaic knowledge bank; (2) mitigating the gap between formulaic knowledge and specific schema via improving the grounding model; (3) driving the parser to fully make use of more complicated (e.g., commonsense) formulaic knowledge.