C ollo QL : Robust Text-to- SQL Over Search Queries (2020)

Abstract: Translating natural language utterances to executable queries is a helpful technique in making the vast amount of data stored in relational databases accessible to a wider range of non-tech-savvy end users. Prior work in this area has largely focused on textual input that is linguistically correct and semantically unambiguous. However, real-world user queries are often succinct, colloquial, and noisy, resembling the input of a search engine. In this work, we introduce data augmentation techniques and a sampling-based content-aware BERT model (ColloQL) to achieve robust text-to-SQL modeling over natural language search (NLS) questions. Due to the lack of evaluation data, we curate a new dataset of NLS questions and demonstrate the efficacy of our approach. ColloQL’s superior performance extends to well-formed text, achieving an 84.9% (logical) and 90.7% (execution) accuracy on the WikiSQL dataset, making it, to the best of our knowledge, the highest performing model that does not use execution guided decoding.

## introduction relational databases store a vast amount of the world's data and are typically accessed via structured query languages like sql. a natural language interface to these databases (nlidb) could significantly improve the accessibility of this data by allowing users to retrieve and utilize the information without any programming expertise. with the release of large-scale datasets @xcite @xcite , this task has gained a lot of attention and has been widely studied in recent years. prior research has primarily focused on translating grammatical, complete sentences to queries. however, an internal user survey on the search service of a major customer relationship management (crm) platform foot_0 revealed that users have a tendency to communicate in a colloquial form which could vary from using only keywords ("player 42") to very short phrases ("show player 42") to complete sentences ("who is the player who wears jersey 42?"). apart from variation in style, users dropping content words from their searches in the interest of brevity also has the potential consequence of making their questions ambiguous. this could render the task unsolvable even to models accustomed to the nls style of text. for example, in figure 1 , without the word "jersey", it is impossible to identify which column's value (id or jersey) must equal 42. in this work, we show that text2sql systems trained on only complete sentences struggle to adapt to the noisy keyword/short phrasal style of questions. to combat this, we introduce different data augmentation strategies inspired from our user search patterns and style. to tackle the induced ambiguity, a potential solution is to utilize the table content by allowing the model to scan the table for different terms present in the question and utilize that information to disambiguate (if the token "42" was only found in the jersey column, then jersey must be the column equal to 42). though effective, this approach could become prohibitively expensive (in terms of inference time or memory required) on large tables as the model would have to search over the entire of the table content for every question. we hypothesize that in most cases, the model only needs samples from the table content and not the exact rows that match tokens in the nls question to disambiguate columns. for example, if the id column contained alpha-numeric ids, player and nationality contained strings, and jersey contained two digit numbers, then jersey must be the column equal to 42. sampling alleviates the need of a full table scan for every question. the samples for each column could be generated offline and remain unchanged across questions or periodically refreshed (to reflect potential distribution shifts in the table or user queries), allowing for adaptation and personalization without retraining the model. in summary, our contributions are as follows: apart from adapting to nls style questions, colloql also achieves state-of-the-art performance on the original wikisql @xcite dataset, outperforming all baselines that do not use execution guided decoding. we base our work off sqlova @xcite but our methods are generalizable to other approaches foot_1 . seq2seq models frame the task as an encoderdecoder problem by trying to generate the sql query token-by-token from the input question. however, as noted by @xcite these models suffer from the "order matters" issue where the model is forced to match the ordering of the where clauses. @xcite employ reinforcement learning based method to overcome this issue but the gains from this has been limited as noted in @xcite . seq2tree models generate the sql query as an abstract syntax tree (ast) instead of a token sequence @xcite . these approaches define a generation grammar for sql and learn to output the action sequence for constructing the ast @xcite . seq2tree approaches are widely adopted for benchmarks that contain complex sql queries @xcite as the syntactic constraints they adopt are effective at pruning the output search space and capturing structural dependencies. however, they do not show much advantage on the wikisql benchmark where the sql asts are largely flat. sqlnet @xcite introduces the concept of a sql-sketch, where it generates a sketch capturing the salient elements of the query as opposed to directly generating the query itself. sqlnet uses lstms to encode the question and headers and employs column attention to predict different components of the sql-sketch. as shown in figure 2 , the query is decomposed into different components which are predicted individually. type-sql @xcite extends upon this approach by augmenting each token in the question with its type (whether it resembles the name of the column, freebase entity type, etc). sqlova @xcite replaces the lstms encoder from sqlnet and uses bert to encode the question and headers jointly. unlike sqlnet, sqlova does not share any parameters in the decoders and identifies the where clause values using span detection instead of pointer generators. hydranet @xcite breaks down the problem into column-wise ranking and decoding and assembles the outputs from each column to create the sql query. text-to-sql with table content recent works like nl2sql-rule @xcite , rat-sql @xcite and photon @xcite have looked into incorporating table content into the sql generation. nl2sql-rule augments bert representations with mark vectors for each question and table header token to indicate a match across the two parts. photon only incorporates the content of a limited set of categorical fields when there is an exact match with a question token. unlike nl2sql-rule, colloql includes table content in the bert encoder allowing it to form content-enhanced question and header representations and unlike photon, colloql incorporates content for all columns and includes samples even when there is not an exact match to disambiguate columns effectively. tabert @xcite lifted the idea further by pre-training joint representation of text and table taking into account row subsampled in a random or relevance-based manner. the pre-trained joint representation has been shown to outperform vanilla language models in several table qa and semantic parsing tasks. text-to-sql with execution guided decoding one common theme across all the high performing models on wikisql is that they all employ execution guided (eg) decoding. first introduced by @xcite , eg is a technique where partial sql queries are executed and their results are used to guide the decoding process. while eg has been shown to boost accuracy significantly, we do not apply execution guided decoding on our models for two reasons: firstly, most eg methods modify the predicted query based on whether an empty set is returned. while this works well in the wikisql setting, having no results is often not due to an erroneous query. it is not uncommon for users to issue searches like "my escalated support cases"(with the expectation of surfacing zero records) or "john doe leads"(to ensure that a record does not already exist before creating one) and we wanted to eliminate the reliance on database outputs to translate a query correctly. secondly, database tables could have over 1m records and performing multiple database executions for every query could be expensive and is not always feasible whilst keeping up with the latency requirements of clients. text-to-sql with noisy user input while recent text-to-sql research typically focus on benchmark datasets with complete and grammatical input, noisy user queries are commonly encountered in practical nlidbs. previous work have proposed several ways to address this issue. @xcite introduced non-standard combinators to a combinatorial categorical grammar (ggg) based semantic parser to handle flexible word order and telegraphic language. @xcite and @xcite developed interactive semantic parsing models that generate clarification questions for user to complete their underspecified queries. @xcite paraphrases an ambiguous input into a less ambiguous form. @xcite generates default logical forms for underspecified input. @xcite synthesized a new dataset and trained question filter to identify noisy user input and prompt user to rephrase. our work focus on handling short user utterances typically found in the search service of salesforce crm, where sampling-based content-aware models are effective at resolving most ambiguities. ## task and datasets the text2sql task is to generate a sql query from a natural language question and the database schema/content. in this work, we use the wik-isql dataset @xcite as it most closely matches the queries we expect to serve in a crm. our users typically don't issue linguistically complex queries requiring joins or nesting but instead focus on filtering a single table based on certain clauses. wikisql contains over 80k natural language questions distributed across 24k tables and their gold sql queries. the performance is typically evaluated on two different types of accuracies -logical form (lf) and execution (ex). lf measures if the generated query exactly matches the gold query while ex executes the predicted and gold queries on the database and verifies if the answers returned by both are equal. note that lf is a stricter metric as many different sql queries could produce the same output. the wikisql dataset mostly comprises of verbose questions which differ in style as compared to the nls questions issued by our users. synthesizing user utterances from gold sql labels since wikisql contains the gold labels for the sql sketch, we can use this data to generate nls-style questions. by analyzing our user search queries (which resemble those shown in original query -who is the player of australian nationality that wears jersey number 42? generated queriesplayer jersey 42 australian nationality; 42 jersey australian nationality player; australian nationality jersey 42 player; . . . supporting relational symbols in user utterance we identify popular query ngrams when the conditional operator in the sql-sketch corresponds to either ">" or "<" and randomly replace these ngrams ("bigger than", "larger than", etc) with the operator symbols, allowing our model to properly interpret them. controlled question simplification since wik-isql contains no keyword-based questions and only a small portion of questions that are succinct enough to require reasoning over the table content, we employ a sentence simplification model followed by manual verification to create a test dataset to evaluate performance on nls questions. a common user behavior is to drop unnecessary words from complete sentences to create shorter questions. we simulate this behavior by simplifying/compressing sentences to reduce verbosity. note that keyword queries can be viewed as an extreme case of sentence simplification where only the required keywords are retained. we make use of the controllable sentence simplifier by @xcite to compress sentences to a desired length whilst retaining a specified set of keywords. we specify the list of keywords to be the header name of the select column, the values in the where columns (we ignore the header names for the where columns as users tend to omit them from their queries). in total, we create two datasets: short questions with gold sql labels and replacement of relation symbols, and simple questions with controlled sentence simplification. manually verified test set we create a highquality test set by manually verifying a subset of simple questions foot_2 . a potential problem with sentence simplification models is ensuring that the shortened version still has enough information to execute the query correctly. this could vary based on the table content and is difficult to identify if the query is impossible to be executed correctly. we had a team of data scientists and engineers proficient in sql to verify/correct outputs produced by the sentence simplification model and generated 400 queries for testing. we show examples in this dataset and report our manual quality evaluation in § a.1. ## proposed approach following @xcite and @xcite , we decompose the sql generation task into 6 different subtasks -one for each component of the sql-sketch. these subtasks all share a common encoder but use different decoder layers. the encoder is a bert model @xcite which produces contextualized representations of the question, headers and the decoders largely use a task-specific lstm with column-attention. column-attention @xcite is a mechanism where each header attends over all query tokens to produce a single representation over which a dense layer is used to predict probabilities. the select, aggregation, where-num, and whereoperator branches use lstms + column-attention followed by a softmax layer to output probabilities. the where-column branch is similar but uses a sigmoid instead as multiple columns could appear in the where clause and the where-value outputs start-end spans for the values from the question. figure 3 highlights the architecture of our model. we retain the same encoder-decoder architecture as sqlova as our main contribution lies in the data augmentation and content sampling techniques to handle nls questions. ## experiment setup we use the base version of bert in all our experiments and made necessary changes for sampling on the original sqlova codebase. we use adam (kingma and ba, 2019) optimizer with a learning rate of 1e-3 for the decoder layers and 1e-5 for the bert model. ## error analysis we classified the errors made by our model on the colloql curated dataset into two major categories: aggregation -given that wikisql contains noisy labels for aggregation component @xcite and the model was optimized for accuracy on wikisql, there are some errors in predicting this slot. select columns -the simplified questions are often more ambiguous when predicting whether a column is a target to be selected or is used in a filtering condition (e.g. for the question "smallest tiesplayed 6 years", the model interprets it as select min(years) where @xmath0. additionally, we noticed that our annotators simplified column headers like "shortstop" and "rightfielder" to "ss" and "rf", making the question very difficult to solve. ## conclusion and future work in this work we tackled the task of converting noisy (short, potentially ambiguous) search-like (nls) questions to sql queries. we introduced data augmentation strategies to adapt to the nls style of text and a novel content enhancement to bert via two sampling strategies -random and relevance sampling. random sampling overcomes some of the performance / privacy challenges of incorporating table content and relevance sampling achieves state-of-the-art performance when access to table content is permitted. finally, we also curated a new held-out dataset to evaluate performance against nls questions. in the future, we hope to explore different sampling techniques (based on user history, sampling to maximize discernment between columns) to enhance performance. besides, our approach and dataset mainly target telegraphic queries that can be effectively disambiguated with table contents, which frequency occur in our search service. we plan to extend our work to handle other types of input ambiguities and other application domains.