Enforcing Consistency in Weakly Supervised Semantic Parsing (2021)

Abstract: The predominant challenge in weakly supervised semantic parsing is that of spurious programs that evaluate to correct answers for the wrong reasons. Prior work uses elaborate search strategies to mitigate the prevalence of spurious programs; however, they typically consider only one input at a time. In this work we explore the use of consistency between the output programs for related inputs to reduce the impact of spurious programs. We bias the program search (and thus the model’s training signal) towards programs that map the same phrase in related inputs to the same sub-parts in their respective programs. Additionally, we study the importance of designing logical formalisms that facilitate this kind of consistency-based training. We find that a more consistent formalism leads to improved model performance even without consistency-based training. When combined together, these two insights lead to a 10% absolute improvement over the best prior result on the Natural Language Visual Reasoning dataset.

## introduction semantic parsers map a natural language utterance into an executable meaning representation, called a logical form or program @xcite . these programs can be executed against a context (e.g., database, image, etc.) to produce a denotation (e.g., answer) for the input utterance. methods for training semantic parsers from only (utterance, denotation) supervision have been developed @xcite @xcite ; however, training from such weak supervision is challenging. the parser needs to search for the correct program from an exponentially large space, and the presence of spurious programs-incorrect repre-figure 1 : utterance x and its program candidates z 1 -z 4 , all of which evaluate to the correct denotation (true). z 2 is the correct interpretation; other programs are spurious. related utterance x shares the phrase yellow object above a black object with x. our consistency reward would score z 2 the highest since it maps the shared phrase most similarly compared to z . in this work we encourage consistency between the output programs of related natural language utterances to mitigate the issue of spurious programs. consider related utterances, there are two boxes with three yellow squares and there are three yellow squares, both containing the phrase three yellow squares. ideally, the correct programs for the utterances should contain similar sub-parts that corresponds to the shared phrase. to incorporate this intuition during search, we propose a consistencybased reward to encourage programs for related utterances that share sub-parts corresponding to the shared phrases ( §3). by doing so, the model is provided with an additional training signal to distinguish between programs based on their consistency with programs predicted for related utterances. we also show the importance of designing the logical language in a manner such that the groundtruth programs for related utterances are consistent with each other. such consistency in the logical language would facilitate the consistency-based training proposed above, and encourage the semantic parser to learn generalizable correspondence between natural language and program tokens. in the previously proposed language for the natural language visual reasoning dataset (nlvr; @xcite , we notice that the use of macros leads to inconsistent interpretations of a phrase depending on its context. we propose changes to this language such that a phrase in different contexts can be interpreted by the same program parts ( §4). we evaluate our proposed approaches on nlvr using the semantic parser of @xcite as our base parser. on just replacing the old logical language for our proposed language we see an 8% absolute improvement in consistency, the evaluation metric used for nlvr ( §5). combining with our consistency-based training leads to further improvements; overall 10% over the best prior model, reporting a new state-of-the-art on the nlvr dataset. ## background in this section we provide a background on the nlvr dataset @xcite and the semantic parser of @xcite . ## consistency reward for programs consider the utterance @xmath0. there are many program candidates decoded in search that evaluate to the correct denotation. most of them are spurious, i.e., they do not represent the meaning of the utterance and only coincidentally evaluate to the correct output. the semantic parser is expected to distinguish between the correct program and spurious ones by identifying correspondence between parts of the utterance and the program candidates. consider a related utterance @xmath1. the parser should prefer programs for x and x which contain similar sub-parts corresponding to the shared phrase @xmath2. that is, the parser should be consistent in its interpretation of a phrase in different contexts. to incorporate this intuition during program search, we propose an additional reward to programs for an utterance that are consistent with programs for a related utterance. specifically, consider two related utterances x and x that share a phrase p. we compute a reward for a program candidate z of x based on how similarly it maps the phrase p as compared to a program candidate z of x . to compute this reward we need (a) relevant program parts in z and z that correspond to the phrase p, and (b) a consistency reward that measures consistency between those parts. our semantic parser (from krishnamurthy et al. ( 2017 )) outputs a linearized version of the program @xmath3, . . . , z t ], decoding one action at a time from the logical language. at each time step, the parser predicts a normalized attention vector over the tokens of the utterance, denoted by [a t 1 , . . . , a t n ] for the z t action. here, n @xmath4, n ]. we use these attention values as a relevance score between a program action and the utterance tokens. given the phrase p with token span [m, n], we identify the relevant actions in z as the ones whose total attention score over the tokens in p exceeds a heuristically-chosen threshold τ = 0.6. a(z, p) = z t @xmath5, t ] and n @xmath6 this set of program actions a(z, p) is considered to be generated due to the phrase p. for example, for utterance there is a yellow object above a black object, with program objexists(yellow(above(black(allobjs))), this approach could identify that for the phrase yellow object above a black object the actions corresponding to the functions yellow, above, and black are relevant. as we do not know the gold program for x , we decode top-k program candidates using beamsearch and discard the ones that do not evaluate to the correct denotation. we denote this set of programs by z c . now, to compute a consistency reward c(x, z, x ) for the program z of x,we take a weighted average of s(z, z ) for different @xmath7. consistency reward based parser given x and a related utterance x , we use c(x, z, x ) as an additional reward in eq. 1 to upweight programs for x that are consistent with programs for x . this consistency-based reward pushes the parser's probability mass towards programs that have consistent interpretations across related utterances, thus providing an additional training signal over simple denotation accuracy. the formulation presented in this paper assumes that there is a single related utterance x for the utterance x. if multiple related utterances are considered, the consistency reward c(x, z, x j ) for different related utterances x j can be summed/averaged to compute a single consistency reward c(x, z) the program z of utterance x based on all the related utterances. ## consistency in language the consistency reward ( §3) makes a key assumption about the logical language in which the utterances are parsed: that the gold programs for utterances sharing a natural language phrase actually correspond to each other. for example, that the phrase yellow object above a black object would always get mapped to yellow(above(black)) irrespective of the utterance it occurs in. on analyzing the logical language of @xcite , we find that this assumption does not hold true. let us look at the following examples: x 1 : there are items of at least two different colors z 1 : objcolorcountgrteq(2, allobjs) x 2 : there is a box with items of at least two different colors z 2 : boxexists( membercolorcountgrteq(2, allboxes)) here the phrase items of at least two different colors ## conclusion we proposed two approaches to mitigate the issue of spurious programs in weakly supervised semantic parsing by enforcing consistency between output programs. first, a consistency based reward that biases the program search towards programs that map the same phrase in related utterances to similar sub-parts. such a reward provides an additional training signal to the model by leveraging related utterances. second, we demonstrate the importance of logical language design such that it facilitates such consistency-based training. the two approaches combined together lead to significant improvements in the resulting semantic parser.