Improving Text-to- SQL Semantic Parsing with Fine-grained Query Understanding (2022)

Abstract: Most recent research on Text-to-SQL semantic parsing relies on either parser itself or simple heuristic based approach to understand natural language query (NLQ). When synthesizing a SQL query, there is no explicit semantic information of NLQ available to the parser which leads to undesirable generalization performance. In addition, without lexical-level fine-grained query understanding, linking between query and database can only rely on fuzzy string match which leads to suboptimal performance in real applications. In view of this, in this paper we present a general-purpose, modular neural semantic parsing framework that is based on token-level fine-grained query understanding. Our framework consists of three modules: named entity recognizer (NER), neural entity linker (NEL) and neural semantic parser (NSP). By jointly modeling query and database, NER model analyzes user intents and identifies entities in the query. NEL model links typed entities to schema and cell values in database. Parser model leverages available semantic information and linking results and synthesizes tree-structured SQL queries based on dynamically generated grammar. Experiments on SQUALL, a newly released semantic parsing dataset, show that we can achieve 56.8% execution accuracy on WikiTableQuestions (WTQ) test set, which outperforms the state-of-the-art model by 2.7%.

## introduction as a natural language interface to database, textto-sql semantic parsing has made great progress in recent years with availability of large amount of annotated data and advances of neural models @xcite @xcite @xcite . these models typically employ a standard encoderdecoder modeling paradigm where model first encodes query and schema, then autoregressively decodes an executable program which could be a se-quence of logical form tokens for flat decoding @xcite or an abstract syntax tree (ast) for structured decoding @xcite . either way, columns and tables are copied from input schema and literal values are copied from input query using pointer network in output program @xcite . despite the success of these models, there are several issues that are left unaddressed for real applications. first, without fine-grained query understanding, autoregressive top-down decoding suffers from generalizing to unseen query patterns at inference time @xcite @xcite , a problem commonly known as compositional generalization. specifically, model may fail to synthesize correct sql for compound input queries like "where is the cyclist who has the most points from" given training queries such as "where is the runner from", "which cyclist gets the most medals". second, in previous works literal values in output logical forms are either omitted @xcite or directly copied from input utterances @xcite . the former will generate non-executable queries. the latter is problematic because mentions in query are often different from their canonical forms in database. for instance, assuming a query like "how many points does lbj get in last game", directly copying word "lbj" into sql query won't match the name "le-bron james" in the database. third, as is shown in @xcite ; @xcite , structured decoding is effective in semantic parsing tasks as it's more likely to generate coherent, executable sql queries. in structured decoding, a sequence of production rules is generated from context free grammars along with schema and literal values. however, among existing solutions, some parsers require manually designed grammars @xcite . others like rat-sql in wang et al. (2020) use grammar generated from compiler tool which is arxiv:2209.14415v1 [cs.cl] 28 sep 2022 often redundant, opaque to understand and offering no flexibility in model design. to address these challenges, we propose a robust, unified framework to solve text-to-sql problem. inspired by a recent work @xcite , the foundation of our parser is based on fine-grained query understanding. we leverage a span based named entity recognition (ner) model to chunk input query and extract sql-typed entities. based on the type information we link entities to database using neural entity linker (nel) model. nel provides linked literal values to the parser, thus the generated sql is executable. the final module of our framework is a grammar-based seq2seq parser which synthesizes executable logical forms from natural language query (nlq), schema, linking results and grammar. in our parser, we dynamically build logical form grammars from training data. this approach frees us from manually constructing grammars and streamlines development of parser model. at the same time, as grammar creation is agnostic to database, our framework is more general-purpose comparing with previous works. in addition to nlq and schema, linking information from nel is also fed to the parser to ensure global reasoning in the decoding. concretely this linking feature will help guide model to select proper actions in decoding. a recent work @xcite takes a similar approach as our framework -they have an extractor model to extract entities and then link mentions to database. however, they use entity label relations to construct logic forms which greatly limits complexity of generated program. by decoupling query understanding, linking and parsing, our framework offers better explainability and flexibility in model design and optimization. a major challenge to build such pipeline system is fine-grained annotations which are needed to train entity recognition model and entity linking model. to tackle this issue, we leverage the newly released squall dataset which provides alignment annotations between nlq tokens and logical forms @xcite . instead of using alignment signal as attention supervision as in @xcite and @xcite , we programmatically convert the alignment annotations to entity annotations and linking annotations, and use these supervision signals to train ner and nel models. a training example is shown in figure 1 that illustrates how the aforementioned conversion works. we evaluate our framework on squall dataset which has fine-grained alignment annotations. on the dev set, our framework obtains 49.36% logical form exact match (em) accuracy and 69.14% execution accuracy, which is 2.2% and 2.6% improvement comparing with the best model in squall paper, respectively. ## approach in this section, we describe our pipeline framework and its application to chunking, linking and parsing tasks. ## conclusion in this work, we proposed a novel, general-purpose text-to-sql semantic parsing framework which is based on fine-grained query understanding. the framework tackles several pain points in the textto-sql problem and offers a new robust approach for real-life applications. our framework outperforms previous state-of-the-art result by 2.7% on squall test set. in the future, we plan to explore using fine-grained query understanding results to constrain decoding search space in parser to further improve system performance.