An efficient method for Natural Language Querying on Structured Data (2023)

Abstract: We present an efficient and reliable approach to Natural Language Querying (NLQ) on databases (DB) which is not based on text-to-SQL type semantic parsing. Our approach simplifies the NLQ on structured data problem to the following “bread and butter” NLP tasks: (a) Domain classification, for choosing which DB table to query, whether the question is out-of-scope (b) Multi-head slot/entity extraction (SE) to extract the field criteria and other attributes such as its role (filter, sort etc) from the raw text and (c) Slot value disambiguation (SVD) to resolve/normalize raw spans from SE to format suitable to query a DB. This is a general purpose, DB language agnostic approach and the output can be used to query any DB and return results to the user. Also each of these tasks is extremely well studied, mature, easier to collect data for and enables better error analysis by tracing problems to specific components when something goes wrong.

## introduction with the recent revolution in information retrieval and question answering, powered by deep learning models, asking queries in a more natural question format e.g. who scored the most points in the nba back in 2018? have become commonplace instead of keyword based searches. more recently models like chatgpt 1 directly generate responses instead of just highlighting text in webpages. a large majority of work in large scale qa systems has been on documentqa @xcite , wherein both the query and the retrieval unit is of text modality. here, both query and documents are generally embedded into a vector representation (generally in the same d-dimensional space) and fast maximum inner product search is used to retrieve the top documents. similar approaches have been used for 1 equal contribution. 1 https://openai.com/blog/chatgpt/ image search @xcite . however, when the information to be retrieved is in structured form i.e. table or group of tables in a database; dual embedding approaches are less common. here, semantic parsing i.e. translating the natural language query into a formal meaning representation e.g. sql are more common. this has inspired several text-to-sql approaches @xcite ; @xcite ; @xcite ; @xcite . @xcite introduced spider, a large-scale complex and crossdomain semantic parsing and text-to-sql dataset. models are evaluated on (clause level) exact match between the gold sql query and the generated one, and the execution accuracy. however, building such models have many practical constraints, perhaps the most important one being collection of domain specific annotated data. annotators not only need to be well versed in the query language but also have detailed knowledge of the comprehensive database schema, table structure etc. it is one thing to know a unique list of all the tables and/or columns in a database, but it is even more demanding to remember which schema or table they belong to and have to refer to this everytime to manually write an output query (even with templates provided). additionally, if the schema/table structure changes with columns added/dropped/moved from one table, then some queries might become invalid. furthermore, the models themselves can suffer from some of the common issues associated with auto-regressive generative models such as repetition, hallucination etc. although these can to some extent be mitigated via decoding constraints, the process is still cumbersome and the gains in most practical use cases might not be worth it. finally, if the db type is changed and uses some other query language then annotating newer data might require re-training the annotation team and even existing queries would need to be translated into the second language, which might be difficult for certain language pairs. an alternative end-to-end approach first introduced in @xcite has also been explored in the literature where the question and flattened table are jointly embedded into a transformer model and trained using masked language modeling (mlm) to table cells. while this approach can work well on smaller tables and documents with text and embedded tables, it would be difficult to scale to large tables and have even bigger controllable generation issues than semantic parsing approaches. in this paper, our primary contribution is proposing a simple yet powerful and configurable framework for a reliable question answering system on structured data by converting the multi-domain nlq on db problem to (1) domain prediction (2) multihead slot tagging (3) raw slot value resolution or disambiguation and (4) deterministic algorithm to convert the outputs of the above three into a query for given database. for (1) and ( 2 ) we use elemental nlp models for text classification and token classification (like ner), respectively. for slot value resolution we propose a suite of methods depending on the data type of the slot extracted and intrinsic nature of the problem (lexical vs semantic similarity; contextual vs non-contextual resolutions). this approach allows us to annotate data much faster as annotators only need to know the names of the tables and unique set of columns across all tables. furthermore, we can also individually test, improve and troubleshoot potential defects to particular components of the pipeline as required; something critical in real world production systems. we posit that our approach can be applicable to a large majority of business use cases where the natural language queries being asked do not need overly complex sub-querying and joins. our method is capable of scaling to when the number of total unique columns (which is equivalent to slot types) is of the order of tens to low hundreds and same for number of tables/domains. in terms of queries we are able to theoretically support selection, filtering, sorting and aggregation and joins (in a limited capacity as described below). ## conclusion we presented a simple, yet effective and highly configurable framework for natural language querying on structured database tables which circumvents