Conversing with databases: Practical Natural Language Querying (2023)

Abstract: In this work, we designed, developed and released in production DataQue – a hybrid NLQ (Natural Language Querying) system for conversational DB querying. We address multiple practical problems that are not accounted for in public Text-to-SQL solutions – numerous complex implied conditions in user questions, jargon and abbreviations, custom calculations, non-SQL operations, a need to inject all those into pipeline fast and to have guaranteed parsing results for demanding users, cold-start problem. The DataQue processing pipeline for Text-to-SQL translation consists of 10-15 model-based and rule-based components that allows to tightly control the processing.

## introduction large amount of companies' data are stored in relational databases -operational data, markets' data, clients data. these data are critical for decision making, however the most common channel for decision-makers to get a view of these data are (semi-) regular reports generated by data analysis professionals. quick hypotheses validation is rarely, if ever, possible for majority of nontechnical business stakeholders. thus, one of the most valuable assets of a company -its data -often appears to be "locked" in the company's databases. another common interface to dbs for nontechnical personnel are data dashboards. however, they have limitations -set of data views is usually fixed, not allowing for custom ones without writing structured queries; the ui of a dashboard tends to grow complex very quickly as business users ask developers to add more and more views, it becomes hard to navigate and hard to learn for new users; adding a new view to a dashboard requires developer work and takes time. a solution to the problem are the so called natural language querying systems with a text-to-sql models @xcite , @xcite or other text-to-query model at the core. nlq allows non-technical users to formulate data requests as natural language questions. for example, "show me sales last quarter by region". there is a lot of focus on nlq recently in the industry, both large and specialized vendors offer nlq solutions: qlik, tableau, powerbi, ibm watson, amazon quicksight, google bigquery, tellius, borealis, and dozens others. there are also multiple opensource models and frameworks for text-to-sql conversion trained on public datasets. there are two critical factors that limit usefulness of most vendor and open-source nlq solutions: let us use as a running example one of our internal applications -a database of daily profit and losses for thousands of financial trading desks in countries across the globe organized into a multilevel multi-dimensional taxonomy of businesses. consider a typical and a relatively simple user question in this usecase: "yoy emea prime ytd pnl + forecast". first we can notice it barely resembles proper english. trading floor language is often very compressed and full of jargon. public nlq systems trained on proper english inputs would be mostly useless on such questions. unpacking this question gives us "over the past 5 years, in every year, take same dates range as between the first day of the year and the current date, select p&l for prime finance trading desks in emea region countries and concatenate these values with full year forecasted p&l values". this is something that public nlq systems could parse, but, unfortunately, real business users never formulate questions like this. one could think it's possible to solve this with a proper query expansion logic -i.e. making all the above substitutions in the question itself. however, the business logic in many cases depends on the 1) the presence or absence of certain entities, e.g. if "moving avg" is mentionedthis implies different default date range 2) on the value of entities, e.g. implied conditions might be different for different trading desks and 3) on the db table that user is asking about. so even to expand the question, we first need to parse it and convert to sql, thus the text-to-sql model should work on these "short" questions. then the business logic is applied on the parsed question and at different stages of parsing -some at ner/ned stage, some at intent understanding, some at sql query construction, etc. see section 4.1 for statistics of sample user questions. in the section 2 we review some related work, in section 3 we review the system architecture and in section 4 we provide some testing results. ## related work text-to-sql translation has become an active research topic for a number of years. solutions at early stage were more domain-specific, and often adopted rule-based approaches @xcite . in recent years, several public data sets, including wikisql @xcite , spider @xcite , and cosql @xcite , became available, which contain hundreds of databases from diverse domains. effort is shifting towards building domain-agnostic generic model with deep neural networks. the structure of neural networks model evolved from general purpose sequence-to-sequence model, to sequenceto-sketch model @xcite , and normally employs encoder and decoder structure. on the encoder side, techniques such as relation-aware schema encoding @xcite has been developed to help link tokens in question to database schema. on the decoder side, different techniques around constrained decoding, such as picard @xcite , has been developed to improve decoder performance. instead of decoding directly into sql statement, it's easier to decode into an intermediate representation of sql. different ways of generating intermediate representation of sql have been proposed in rat-sql @xcite , syntax-sql @xcite , nat-sql @xcite . another option is to decode into an abstract syntax tree (ast) @xcite . inspired by sequence-to-sketch model and the ast approach, we designed and developed an nlu parsing pipeline and slot-based sql generation engine, as well as a custom ast that can be used for sql generation, as well as for modification of sql based on business rules. as training of textto-sql model requires large amount of labelled data, various data augmentation methods have been proposed and tested @xcite . we also developed data augmentation tool using techniques, such as entity swapping, and paraphrasing. most recent advancements in the field are around leveraging capability of generative large language models for text-to-sql translation through prompt engineering and in context learning @xcite . ## discussion the overall accuracy of the system is relatively high at 88%, however, production requirement is that users never unknowingly get incorrect data from the system. error analysis indicated that the 12% of cases when the system didn't return the expected data are split approximately 30%/70% between 1) system returning some data and 2) system returning a text message (e.g, out-of-domain classifier recognized the question as ood or intent classifier mis-recognized the intent). this means, in 12%•30% ≈ 4% cases, the system returns data that doesn't match the user's question. further error analysis indicated that in almost all of these cases, the system was not able to recognize or resolve one of the entities that user has mentioned. to address these situations, we take "human in the loop" path and show the user the parsing results in a simplified form, see fig. 3 . this allows users to recognize when the system incorrectly parsed their question and returned them not the data they asked. ## conclusions we designed, developed and deployed in production a hybrid conversational nlq system for several real-world usecases in a large international financial institution. the approach allows to address multiple conflicting practical requirementscustom domain language with jargon and abbreviations, numerous complex implied conditions in the users' questions, need to incorporate new business-rules into the model pipeline quickly as user feedback comes and have a guaranteed behavior of the system, high accuracy and interpretability of the results. vendor solutions or open-source end-to-end text2sql models do not allow for required level of customization and controllability for heavily domain-specific applications. we believe, our hybrid framework strikes a good practical balance between leveraging pre-trained language understanding models and rules.