Neural Approaches for Natural Language Interfaces to Databases: A Survey (2020)

Abstract: A natural language interface to databases (NLIDB) enables users without technical expertise to easily access information from relational databases. Interest in NLIDBs has resurged in the past years due to the availability of large datasets and improvements to neural sequence-to-sequence models. In this survey we focus on the key design decisions behind current state of the art neural approaches, which we group into encoder and decoder improvements. We highlight the three most important directions, namely linking question tokens to database schema elements (schema linking), better architectures for encoding the textual query taking into account the schema (schema encoding), and improved generation of structured queries using autoregressive neural models (grammar-based decoders). To foster future research, we also present an overview of the most important NLIDB datasets, together with a comparison of the top performing neural models and a short insight into recent non deep learning solutions.

## introduction semantic parsing is the task of mapping natural language (nl) utterances to a formal meaning representation (mr), mainly with the purpose of querying an information source, i.e. a database or a knowledge graph. these machine-readable mrs are sometimes referred to as logical forms (lf). mrs can be expressed using abstract, language-agnostic representations, such as λ-calculus expressions @xcite or abstract meaning representations @xcite . however, they can also be represented directly in a programming or query language, for example as python code @xcite or sql (structured query language) commands @xcite . the text-to-sql semantic parsing task, also known as natural language interfaces to databases (nlidb), has gathered strong research interest even in the first wave of ai @xcite . interest in nlidbs resurfaced in the last five years due to the success of large end-to-end neural architectures for conditional text generation @xcite and the availability of larger datasets @xcite @xcite . early systems adopted pattern-matching, handcrafted grammars and rules, syntactic parsers, and intermediate language representations, which were usually deployed in a multi-stage system @xcite . the main difficulties for the successful application of nlidb systems include: fuzzy linguistic coverage (user does not know when the system truly understands the query), linguistic vs. conceptual failures (nlidb failure caused either by linguistic coverage or the inability to resolve database entities), language ambiguity (e.g. incorrect quantifier scoping, incorrect attachment modifier, nominal compound problem, anaphora resolution, elliptical constructs), and out-of-domain queries. however, recent solutions treat the nlidb problem as a sequence-to-sequence transduction task. in this case, a large end-to-end neural network processes the input sequence (nl utterance x, plus database context c) and generates the output y autoregressively. this approach overcomes the need to learn separate linguistic components and assemble them together. most solutions employ extensions of the seq2seq architecture @xcite , which has replaced multi-stage approaches for a wide range of nlp problems. the objective of this paper is to explore and structure the recent advances in neural solutions for nlidb. although the use of neural networks and deep learning for this task is quite recent, we consider that the current neural nlidb solutions are mature and it is important to acknowledge the most successful contributions in order to plan future research. as this topic is extensive, we do not aim to cover historically important systems that are not using deep learning, but only provide a brief talk about new solutions. for the interested reader, we strongly recommend recent surveys that cover a wide range of non-deep learning systems @xcite . to some extent, our work aims to supplement them with the relevant contributions of neural approaches trained on large datasets. this paper is structured as follows. in section 2 we frame the task of creating a nlidb in relation to the type of data that is available for training and evaluation. in section 3 we provide a detailed description of the key decisions behind the design of current state of the art neural solutions. section 4 grounds the exploration of neural nlidb models with a brief presentation of the latest solutions that do not use deep learning. finally, in section 5 we highlight the most effective innovations that have been introduced recently in the field and how they have affected the performance of nlidb systems. ## methodology neural models have become the standard approach for nlidb solutions. the typical neural architecture consists of an encoder and a decoder component, in a seq2seq approach. in this section, we explore the most significant design considerations for the two components, as presented in recent literature. ## non-deep learning nlidb approaches in this section, we survey recent solutions that do not use deep learning (dl). when it comes to the non-dl nlidb approaches, the main design classes are i) keyword-based; ii) pattern-based; iii) parsingbased; iv) grammar-based @xcite . keyword-based solutions. keyword-based approaches use inverted indexes to map the tokens extracted from natural language to existing concepts in the database and return a query. these approaches cannot express complex query intents accurately or define proper aggregation queries. for example, qaldgen @xcite is a framework that uses the keyword-based approach to construct sparql queries over a knowledge graph for question answering systems. pattern-based solutions. pattern-based methods augment the keyword-based solutions by adding information extracted through basic nlp methods, i.e. stopword removal, part-of-speech tagging, stemming and lemmatization, etc. these approaches are used to extract concepts and aggregation operations from user queries. recipefinder @xcite is a pattern-based system that uses human-computer interaction and basic nlp methods to extract entities and resolve question ambiguity. parsing-based solutions. parsing-based approaches extend pattern-based systems by using advanced nlp techniques to parse and extract the grammatical structure of the questions. nalir @xcite and sqlizer @xcite build intermediate representations of the user question, leveraging a dependency parser and a semantic parser, respectively. the representations are iteratively refined using rules and heuristics and then translated to the target sql. grammar-based solutions. grammar-based systems rely on rules to transparently limit the flexibility of the user question. this approach enables auto-suggestion mechanisms @xcite that guide the user to write only queries that can be correctly interpreted by the system. ## discussion non-dl vs dl approaches. non-dl systems focus on delivering a good user experience through mechanisms such as auto-suggestion and user interaction. however, they typically depend on rules which are tailored for particular domains or databases. this limits their evaluation to relatively small, single-domain datasets (e.g. mas, imdb, yelp). on the other hand, dl solutions have also been successfully applied in recent years, boosted by the availability of large datasets and the flexibility of neural sequence learners, such as the seq2seq architecture. for training and testing, they leverage corpora featuring multiple databases and domains, such as wikisql and spider. while the two strands of work have evolved from different research communities, a common evaluation testbed is required to make the strengths and weaknesses of these lines of work more apparent. comparison of dl approaches. in schema linking. schema linking is a crucial step and provides a significant boost, as highlighted by the ablated vs full rat-sql model (14.4% relative accuracy). the models in the table with no schema linking compensate by a more flexible schema encoding (self-attention in editsql) or a more complex decoder (grammar decoder in syntaxsqlnet). leveraging structure in the database schema. choosing the appropriate model for schema encoding is another important aspect, with more complex approaches, either based on graph neural networks (gnns) or transformers, generally outperforming bidirectional lstms. editsql for instance has a higher em accuracy than the simplest syntaxsqlnet model (36.4% vs 18.9%), even without a grammar-based decoder. this difference mainly comes from the transformer-based schema encoding, which better captures the relationships between question tokens and schema elements. leveraging code structure. the third set of improvements comes from leveraging the grammatical structure of the sql commands via a grammar-aware decoder (either stand-alone or modular). for instance, the simplest syntaxsqlnet model outperforms typesql by a 10.9% margin, likely due to the fact that it is guided by the sql grammar. the bert-free variant of irnet also outperforms gnn @xcite 4.1 typesql @xcite 8.0 syntaxsqlnet @xcite 18.9 syntaxsqlnet @xcite 24.8 syntaxsqlnet @xcite 25.0 gnn @xcite 34.9 gnn @xcite 40.7 gnn @xcite 51.3 gnn + bertrand-dr @xcite 57.9 editsql @xcite 36.4 editsql @xcite 57.6 editsql + bertrand-dr @xcite 58.5 irnet @xcite 53.2 irnet @xcite 61.9 ryansql @xcite 43.4 ryansql @xcite 66.6 rat-sql @xcite 46.2 rat-sql @xcite 60.6 rat-sql @xcite 69.7 (53.2% vs 40.7% accuracy), even though the latter uses a more complex schema encoding. the difference is again likely due to irnet using a grammar-aware decoder. grammar-based decoders benefit even more when simplifying the grammar itself as much as possible. decoding to semql @xcite instead of sql brings an average relative boost of 8.4%. more recently, integrating code idioms into the grammar further boosts performance @xcite . all these examples highlight the importance of simplifying the search space of the decoder by taking advantage of the structural regularities of the sql output. we anticipate this direction to be further exploited by future approaches. ## conclusion in this paper, we have surveyed the most important contributions of deep learning approaches proposed for building nlidb systems. although the focus was on deep learning nlidbs, we have also presented seminal ideas from important classical methods and from recent non-dl solutions. the main conclusion of the paper is that nlidb research has seen a revival in the last years following the release of several large datasets and specific improvements in neural network architectures. the first set of enhancements are task-specific, accounting for structural information either in the database schema (schema encoding) or the sql code (grammar-based decoding). better schema encoding has been achieved with gnns or transformer-based architectures that encode the schema entities as well as relationships between them. syntax-based decoding is still mostly performed with recurrent architectures and further improvements have come from simplifying the sql grammar itself. the second improvement is in line with the recent success of transfer learning in natural language processing, namely leveraging transformer-based contextualized representations via a largely pre-trained bert backbone. this backbone provides better word representations and can be used to encode the question and the database schema more efficiently by attending each other. we expect more sophisticated models that better leverage the structural information of both the database schema and the sql code to dominate future approaches. moreover, as the performance gap on the single-turn task reduces, we also anticipate that the more complex multi-turn task will soon become the testbed of new methods, due to its difficulty and proximity to real world use cases.