SUN : Exploring Intrinsic Uncertainties in Text-to- SQL Parsers (2022)

Abstract: This paper aims to improve the performance of text-to-SQL parsing by exploring the intrinsic uncertainties in the neural network based approaches (called SUN). From the data uncertainty perspective, it is indisputable that a single SQL can be learned from multiple semantically-equivalent questions. Different from previous methods that are limited to one-to-one mapping, we propose a data uncertainty constraint to explore the underlying complementary semantic information among multiple semantically-equivalent questions (many-to-one) and learn the robust feature representations with reduced spurious associations. In this way, we can reduce the sensitivity of the learned representations and improve the robustness of the parser. From the model uncertainty perspective, there is often structural information (dependence) among the weights of neural networks. To improve the generalizability and stability of neural text-to-SQL parsers, we propose a model uncertainty constraint to refine the query representations by enforcing the output representations of different perturbed encoding networks to be consistent with each other. Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms strong competitors and achieves new state-of-the-art results.

## introduction text-to-sql parsing @xcite @xcite aims at converting a natural language (nl) question to its corresponding structured query language (sql) in the context of a relational database (schema). although relational databases equal contribution. † corresponding authors. can be efficiently accessed by skilled professionals via handcrafted sqls, a natural language interface, whose core component relies on text-to-sql parsing, would allow ubiquitous relational data to be accessible to a broader range of non-technical users. therefore, text-to-sql parsing has attracted increasing attention from both academic and industrial communities recently due to its broad applications in question answering, conversational search interaction, and so on. although significant efforts have been devoted to text-to-sql parsing @xcite @xcite @xcite with advanced deep models and architectures by learning black-box mappings between input nl questions and output sqls, there are still several technical challenges for accurate and robust text-to-sql parsing. first, previous models are generally learned to fit the simplified one-to-one mapping relationship, where only one nl question is used as the input, and the appropriate rest are ignored. however, there exists data uncertainty @xcite ) in text-to-sql parsing, i.e., one output sql may correspond to multiple semantically-equivalent nl questions. at inference time, the semantic parser trained on the one-to-one parallel data struggles to deal with adequate variations of the training queries. second, there is often structural information (dependence) @xcite among the weights of neural networks. one challenge in training neural semantic parsers is that such models may overfit the training data since these models only seek a point estimate for their weights, failing to quantify weight (model) uncertainty. for text-to-sql parsing, the model uncertainty brings difficulty in obtaining the encoded representations that can best describe the input data distribution and provide an robust mapping between nl questions and sql queries. to alleviate the aforementioned challenges, in this paper, we propose a generic training approach sun, which explores the data and model uncertainties in text-to-sql parsing. first, we propose a data uncertainty constraint to explore the underlying complementary semantic information among multiple semantically-equivalent queries and learn comprehensive feature representations with strong expressive ability. in particular, we summarize multiple semantically-equivalent source questions into a closed semantic region which is then used to complement the model to generate better sql queries with comprehensive semantics. second, to improve the generalizability and stability of neural text-to-sql parsers, we propose a model uncertainty constraint to refine the query representations by enforcing the output representations of different perturbed encoding networks to be consistent with each other. concretely, we impose the consistency on multiple networks perturbed with dropout for the same input nl question. we summarize our main contributions as follows. (1) we propose a data uncertainty constraint, which aims to explicitly capture comprehensive semantic information among multiple semanticallyequivalent nl questions, and enhance the hidden representations with this complementary information for generating better sql, thus improving the robustness of neural text-to-sql parsers. (2) we employ a model uncertainty constraint to encourage high similarity between the output representations of two perturbed encoding networks for the same input nl question, improving the generalizability and stability of neural text-to-sql parsers. the general idea behind these methods is to construct an encoder to encode the input question together with related table schema and leverage a decoder to generate the target sql based on the output of the encoder. for example, irnet @xcite is a representative neural text-to-sql parser, which leveraged two separate bilstms with selfattention mechanism @xcite to encode the nl question and table schema. subsequently, the graph-based approaches have been proposed, which use relational graph attention networks to deal with the schema entities and question words with structured reasoning. for instance, ratsql @xcite , smbop @xcite and rasap @xcite defined a question-schema graph and employed the relation-aware self-attention mechanism @xcite in the encoding process to jointly learn representations of question words, schema items and edge relations. lgesql @xcite further constructed an edge-centric graph to update the edge features and designed graph pruning to determine the golden schema items related to the nl question. recently, some methods have leveraged the powerful pre-training capabilities of t5 @xcite to generate sql queries. different from graph-based methods, t5-based approaches @xcite adopt the transformer-based architecture for both encoder and decoder and do not need pre-defined graphs, schema linking relations, and grammar-based decoder. pi-card @xcite is a representative t5based method which constrained auto-regressive decoders of language models through incremental parsing. the impressive experimental results verify the ability of the t5-based methods for text-to-sql parsing. uncertainty modeling in nlp uncertainty quantification is an important approach to building robust ai systems. in the field of natural language processing, there are several works @xcite @xcite @xcite @xcite which investigate the effects of quantifying uncertainties in various nlp tasks. for example, @xcite applied a dropout-entropy method to measure uncertainty learning for text classification. @xcite showed that explicitly modeling uncertainties via monte-carlo dropout @xcite could enhance model performances of several nlp tasks. @xcite introduced a series of continuous latent variables to model underlying semantics of source sentences in neural machine translation. @xcite proposed to quantify the confidence of nmt model predictions based on model uncertainty to better cope with noise in synthetic corpora. @xcite considered the intrinsic uncertainty by representing multiple source sentences into a closed semantic region. to our best knowledge, our proposed method is the first attempt to explore uncertainty in text-to-sql parsing. ## conclusion in this paper, we proposed a novel sun method to explore intrinsic uncertainties in text-to-sql parsing. first, we devised a data uncertainty constraint to capture complementary semantic information among multiple semantically-equivalent questions and thus improve the robustness of the text-to-sql parsers. second, a model uncertainty constraint was leveraged to refine the representations by encouraging high similarity between the output representations of two perturbed encoding networks for the same input question. experimental results on five benchmark datasets showed that sun significantly outperformed the compared methods.