Paired Examples as Indirect Supervision in Latent Decision Models (2021)

Abstract: Compositional, structured models are appealing because they explicitly decompose problems and provide interpretable intermediate outputs that give confidence that the model is not simply latching onto data artifacts. Learning these models is challenging, however, because end-task supervision only provides a weak indirect signal on what values the latent decisions should take. This often results in the model failing to learn to perform the intermediate tasks correctly. In this work, we introduce a way to leverage paired examples that provide stronger cues for learning latent decisions. When two related training examples share internal substructure, we add an additional training objective to encourage consistency between their latent decisions. Such an objective does not require external supervision for the values of the latent output, or even the end task, yet provides an additional training signal to that provided by individual training examples themselves. We apply our method to improve compositional question answering using neural module networks on the DROP dataset. We explore three ways to acquire paired questions in DROP: (a) discovering naturally occurring paired examples within the dataset, (b) constructing paired examples using templates, and (c) generating paired examples using a question generation model. We empirically demonstrate that our proposed approach improves both in- and out-of-distribution generalization and leads to correct latent decision predictions.

## introduction developing models that are capable of reasoning about complex real-world problems is challenging. it involves decomposing the problem into subtasks, making intermediate decisions, and combining them to make the final prediction. while many approaches develop black-box models to solve such problems, we focus on compositional structured models as they provide a level of explanation for their predictions via interpretable latent decisions, and should, at least in theory, generalize better in compositional reasoning scenarios. for example, to answer how many field goals were scored in the first half? against a passage containing a footballgame summary, a neural module network (nmn; @xcite would first ground the set of field goals mentioned in the passage, then filter this set to the ones scored in the first half, and then return the size of the resulting set as the answer. learning such models using just the end-task supervision is difficult, since the decision boundary that the model is trying to learn is complex, and the lack of any supervision for the latent decisions provides only a weak training signal. moreover, the presence of dataset artifacts @xcite , and degeneracy in the model, where incorrect latent decisions can still lead to the correct output, further complicates learning. as a result, models often fail to predict meaningful intermediate outputs and instead end up fitting to dataset quirks, thus hurting generalization @xcite . we propose a method to leverage related training examples to provide an indirect supervision to these intermediate decisions. our method is based on the intuition that related examples involve similar sub-tasks; hence, we can use an objective on the outputs of these sub-tasks to provide an additional training signal. concretely, we use paired examples-instances that share internal substructureand apply an additional training objective relating the outputs from the shared substructures resulting from partial model execution. using this objective does not require supervision for the output of the shared substructure, or even the end-task of the paired example. this additional training objective imposes weak constraints on the intermediate out-puts using related examples and provides the model with a richer training signal than what is provided by a single example. for example, what was the shortest field goal? shares the substructure of finding all field goals with how many field goals were scored?. for this paired example, our proposed objective would enforce that the output of this latent decision for the two questions is the same. we demonstrate the benefits of our paired training objective using a textual-nmn @xcite ) designed to answer complex compositional questions on drop @xcite , a dataset requiring natural language and symbolic reasoning against a paragraph of text. while there can be many ways of acquiring paired examples, we explore three directions. first, we show how naturally occurring paired questions can be automatically found from within the dataset. further, since our method does not require end-task supervision for the paired example, one can also use data augmentation techniques to acquire paired questions without requiring additional annotation. we show how paired questions can be constructed using simple templates, and how a question generation model can be used to generate paired questions. we empirically show that our paired training objective leads to overall performance improvement of the nmn model. while each kind of paired data acquisition leads to improved performance, combining paired examples from all techniques leads to the best performance ( §5.1). we quantitatively show that using this paired objective results in significant improvement in predicting the correct latent decisions ( §5.2), and thus demonstrate that the model's performance is improving for the right reasons. finally, we show that the proposed approach leads to better compositional generalization to outof-distribution examples ( §5.3). our results show that paired supervision brings us closer to achieving the stated promise of latent decision models: an interpretable model that naturally encodes compositional reasoning and uses its modular architecture for better generalization. ## paired examples as indirect supervision for latent decisions we focus on structured compositional models for reasoning that perform an explicit problem decomposition and predict interpretable latent decisions that are composed to predict the final output. these intermediate outputs are often grounded in real-world phenomena and provide some explanation for the model's predictions. such models assume that the structured architecture provides a useful inductive bias for efficient learning. for example, for a given input x, a model could perform the computation f (g(x), h(x)) to predict the output y. in this paper, we will need to distinguish between a computation tree and the output of its execution. therefore, we will use the notation z to denote a computation tree, and z to denote the output of its execution. hence we can write, in this work, we propose a method to leverage paired examples-examples whose one or more latent decisions are related to each other-to provide an indirect supervision to these latent decisions. consider paired training examples x i and x j with the following computation trees: these trees share the internal substructure g(x). in such a scenario, we propose an additional training objective s( g(x i ) , g(x j ) ) to enforce consistency of partial model execution for the shared substructure: for example, the two questions on the lhs of figure 1 share the intermediate decision of finding the field goals. i.e., their computation trees share the substructure g(x) = find[field goal]. in such a case, where the outputs of the intermediate decision should be the same for the paired examples, using a similarity measure for s would enforce equality of the latent outputs g(x) . we will go into the specifics of this example in section 3. by adding this consistency objective, we are able to provide an additional training signal to the latent decision using related examples, and hence indirectly share supervision among multiple training examples. to use this consistency objective for x i , we do not require supervision for the latent output g(x i ) , nor the gold end-task output y * j for the paired example x j ; we only enforce that the intermediate decisions are consistent. additionally, we are not limited to enforcing consistency for a single intermediate decision from a single paired example; if x i shares an additional substructure h(x) with a paired example x k , we can add an additional term s ( h(x i ) , h(x k ) ) to eq. 4. our approach generalizes a few previous methods for learning via paired examples. for learning to ground tokens to image regions, @xcite enforce contrastive grounding between the original and a negative token. recently, @xcite use paired utterances to reward semantic parses that map the shared phrase to similar program parts in weakly supervised settings. these are equivalent to using an appropriate s in our frame-work. a few approaches @xcite @xcite ) use an additional objective on model outputs to enforce consistency between paired examples; this is a special case of our framework where s is used on the outputs (y i , y j ), instead of the latent decisions. using paired examples for indirect supervision on latent decisions should be broadly applicable to a wide class of models, and our general formulation of this technique is, we believe, novel. however, the specific application of this method to any particular problem is non-trivial, as work needs to be done to acquire paired data and design a suitable s for the model being studied. in the rest of this work, we present a case study on text-based neural module networks which we believe is promising enough to motivate further applications of this method. ## training via paired examples in neural module networks we apply our approach to improve question answering using a neural module network (nmn; @xcite on the drop dataset @xcite . drop contains complex compositional questions against natural language passages. nmn is a model architecture aimed at reasoning about natural language against a given context (text, image, etc.) in a compositional manner. a nmn maps the input utterance into an executable program representing the compositional reasoning structure required to predict the output. the program is composed of learnable modules that are designed to perform atomic reasoning tasks. for example, to answer @xmath0, a nmn would parse it into a program @xmath1. this program then gets executed by the learnable modules to produce y, essentially performing @xmath2, where @xmath3. given a question q, the gold program z * , and the correct answer a * , maximum likelihood training is used to jointly train the parameters of the modules, as well as a parser that produces the gold program z * . the gold program only supervises the layout of the modules (z in the example in above) and not the outputs of the intermediate modules. that is, we have a * = z * , but no supervision on subparts of z * , such as g(q) . it is extremely challenging to learn the module parameters correctly in the absence of intermediate module output supervision. learning is further complicated by the fact that the space of possible intermediate outputs is quite large and incorrect module output prediction can still lead to the correct answer. for example, the find module in the question above needs to learn to select the spans describing field goals among all possible spans in the passage using just the count value as answer supervision. with no direct supervision for the module outputs, the modules can learn incorrect behavior but still predict the correct answer, effectively memorizing the training data. such a model would presumably fail to generalize. text-nmn we work with the text-nmn of @xcite on a subset of drop which is annotated with gold programs. their model contains find, filter, project, count, find-num, find-date, find-max-num, find-min-num, num-compare, date-compare, num-add, num-diff, time-diff, and spans modules. the find, filter, and project modules take as input an additional question string argument. each module's output is an attention distribution over the relevant support. e.g. find, filter output an attention over passage tokens, find-num over numbers, find-date over dates, etc. given a question q and passage p, bert is used to compute joint contextualized representations for the (question, passage) combination, bert(q, p) ∈ r (|q|+|p|)×d . during execution, the modules that take a question span argument as input (e.g. find) operate on the corresponding slice of this contextualized representation. for example, in @xmath4, to execute find[field goals], the model actually executes find(bert(q, p)[2 : 3]). foot_0 here the slice [2 : 3] corresponds to the contextualized representations for the 2nd through the 3rd token (field goals) of the question. refer gupta et al. (2020a) for details. paired training in nmns we consider a pair of questions whose program trees z share a subtree as paired examples. a shared subtree implies that a part of the reasoning required to answer the questions is the same. since some modules take as input a string argument, we define two subtrees to be equivalent iff their structure matches and the string arguments to the modules that require them are semantically equivalent. for example, subtrees find-num(find[passing touchdowns]) and find-num(find[touchdown passes]) are equivalent, while they are not the same as find-num(find[touchdown runs]) (we describe how we detect semantic equivalence in §4). consider a question q i that shares the substructure g(q) with a paired question q j . since shared substructures are common program subtrees in our case, we encourage the latent outputs, the outputs g(q) of the subtree, to be equal to enforce consistency. as the outputs of modules are probability distributions, enforcing consistency amounts to minimizing the kl-divergence between the two outputs. we therefore maximize the following paired objective from eq. 4, to understand why such an objective is helpful even though the paired examples share exact subtrees, consider the paired examples on the lhs of figure 1 . the substructure g(q) = find[field goal] is shared between them. even though the input string argument to find is the same, what gets executed is g(q i ) = find(bert(x 1 , p)[2 : 3]) and g(q j ) = find(bert(x 2 , p)[7 : 8]), i.e. find gets different contextualized representations of field goal from the two questions. due to different inputs, the output of find could be different, which would lead to inconsistent behavior. our paired objective (eq. 5) would encourage that these two outputs are consistent, thereby allowing sharing of supervision across examples. ## many ways of getting paired data we explore three ways of acquiring paired questions. we show how questions that share substructures can be automatically found from within the dataset ( §4.1), and how new paired questions can be constructed using templates ( §4.2.1), or generated using a question-generation model ( §4.2.2). ## conclusion we propose a method to leverage paired examplesinstances that share internal substructure-to provide a richer training signal to latent decisions in compositional model architectures. we provide a general formulation of this technique which should be applicable to a broad range of models. to validate this technique, we present a case study on text-based neural module networks, showing how to apply the general formulation to a specific task. we explore three methods to acquire paired examples and empirically show that our approach leads to substantially better in-and out-of-distribution generalization of a neural module network in complex compositional question answering. we also show that using our paired objective leads to improved prediction of latent decisions.