On Robustness of Neural Semantic Parsers (2021)

Abstract: Semantic parsing maps natural language (NL) utterances into logical forms (LFs), which underpins many advanced NLP problems. Semantic parsers gain performance boosts with deep neural networks, but inherit vulnerabilities against adversarial examples. In this paper, we provide the first empirical study on the robustness of semantic parsers in the presence of adversarial attacks. Formally, adversaries of semantic parsing are considered to be the perturbed utterance-LF pairs, whose utterances have exactly the same meanings as the original ones. A scalable methodology is proposed to construct robustness test sets based on existing benchmark corpora. Our results answered five research questions in measuring the sate-of-the-art parsers’ performance on robustness test sets, and evaluating the effect of data augmentation.

## introduction semantic parsing aims to map natural language (nl) utterances into logical forms (lfs), which can be executed on a knowledge base (kb) to yield denotations @xcite . at the core of the state-of-the-art (sota) semantic parsers are deep learning models, which are widely known to be vulnerable to adversarial samples @xcite . this kind of examples is created by adding tiny perturbations to inputs but can severely deteriorate model performance. to the best of our knowledge, despite the popularity of semantic parsing @xcite , there is still no published work on studying the robustness of neural semantic parsers against adversarial examples. therefore, we conduct the first empirical study to evaluate the effect of adversarial examples on sota neural semantic parsers. * corresponding author unlike other disciplines, it is unclear what adversaries are for semantic parsers. for computer vision systems, adversaries are often generated by modifying inputs with imperceptible perturbations. in contrast, a flip of single word or a character in an utterance can significantly change its meaning so that the changes are perceptible by humans. to address this issue, @xcite argue that adversaries for sequence-to-sequence models should maximally retain meanings after perturbing inputs. however, any meaningchanging utterances are supposed to have different meaning representations. a robust semantic parser should be invariant to meaning-preserving modifications. in light of this, given a semantic parser, we define its adversaries as the perturbed utterances satisfying two conditions: i) they have exactly the same meanings as the original ones according to human judgements; and ii) the parser consistently produces incorrect lfs on them. although new evaluation frameworks are proposed for nlp tasks @xcite , there is no framework designed for assessing robustness of semantic parsers against meaning-preserving adversaries. the current evaluation metrics focus only on standard accuracy, which measures to what degree predictions match gold standards. as pointed out by @xcite , it is challenging to achieve both high standard accuracy and high robust accuracy, which measures the accuracy on adversarially perturbed examples. in order to facilitate devising novel methods for robust semantic parsers, it is desirable to develop a semantic parsing evaluation framework considering both measures. in this work, we propose an evaluation framework for robust semantic parsing. the framework consists of an evaluation corpus and a set of customized metrics. we construct the evaluation corpus by extending three existing semantic parsing benchmark corpora. in order to generate meaningpreserving examples, we apply automatic methods to modify the utterances in those benchmark corpora by paraphrasing and injecting grammatical errors. among the perturbed examples generated from the test sets, we build meaning-preserving test sets by filtering out the meaning-changing ones using crowdsourcing. the robustness of the semantic parsers is measured by a set of custom metrics, with and without adversarial training methods. we conduct the first empirical study on the robustness of semantic parsing by evaluating three sota neural semantic parsers using the proposed framework. the key findings from our experiments are three-folds: the generated corpus and source code are available at https://github.com/shuo956/onrobustness-of-nerual-smentic-parsing.git 2 related work semantic parsing the sota neural semantic parsers formulate this task as a machine translation problem. they extend seq2seq with attention @xcite to map nl utterances into lfs in target languages (e.g., lambda calculus, sql, python, etc.). one type of such parsers directly generates sequences of predicates as lfs @xcite @xcite . the other type of parsers utilizes grammar rules to constrain the search space of lfs during decoding @xcite @xcite . however, neither of the two types of parsers are evaluated against adversarial examples. ## evaluation framework for robust semantic parsing a robust semantic parser aims to transduce all utterances with or without meaning-preserving perturbations into correct lfs. formally, let @xmath0, . . . , x |x| denote a natural language utterance, and @xmath1, . . . , y |y| be its lf, a semantic parser estimates the conditional probability (denoted by p(y|x)) of an lf y given an input utterance x. a robust parser's predictions are invariant to all x that are generated from x by meaning-preserving perturbations. for any (x, y), let s perturb (x, y) denote the set of all meaning-preserving perturbations of x that is parsed to y: a set of meaning-preserving perburbed examples s perturb (d) w.r.t. a corpus d is the union of all s perturb (x, y) created from d. an adversarial example w.r.t. a semantic parser is an utterance-lf pair, which is in s perturb (x, y) and is parsed into an incorrect lf by that parser. a set of adversarial examples w.r.t. a s perturb (x, y) and a parser f (x) is obtained by: subsequently, an adversary set w.r.t. a semantic parsing corpus d is created by taking the union of all adversary sets created from each example in d. in the following, we present an evaluation framework for robust semantic parsing, which consists of an evaluation corpus, a set of evaluation metrics, and the corresponding toolkit for evaluating any new parsers. ## experiments in this section, we present the first empirical study on robust semantic parsing.