Obtaining Faithful Interpretations from Compositional Neural Networks (2020)

Abstract: Neural module networks (NMNs) are a popular approach for modeling compositionality: they achieve high accuracy when applied to problems in language and vision, while reflecting the compositional structure of the problem in the network architecture. However, prior work implicitly assumed that the structure of the network modules, describing the abstract reasoning process, provides a faithful explanation of the model’s reasoning; that is, that all modules perform their intended behaviour. In this work, we propose and conduct a systematic evaluation of the intermediate outputs of NMNs on NLVR2 and DROP, two datasets which require composing multiple reasoning steps. We find that the intermediate outputs differ from the expected output, illustrating that the network structure does not provide a faithful explanation of model behaviour. To remedy that, we train the model with auxiliary supervision and propose particular choices for module architecture that yield much better faithfulness, at a minimal cost to accuracy.

## introduction models that can read text and reason about it in a particular context (such as an image, a paragraph, or a table) have been recently gaining increased attention, leading to the creation of multiple datasets that require reasoning in both the visual and textual domain @xcite @xcite @xcite @xcite . consider the example in figure 1 from nlvr2: a model must understand the compositional sentence in order to then ground dogs in the input, count those that are black and verify that the count of all dogs in the image is equal to the number of black dogs. unfaithful behaviour of modules, such as multiple reasoning steps collapsing into one, are undesirable in terms of interpretability; when a model fails to answer some question correctly, it is hard to tell which modules are the sources of error. while recent work @xcite ) has shown that one can obtain good performance when using nmns, the accuracy of individual module outputs was mostly evaluated through qualitative analysis, rather than systematically evaluating the intermediate outputs of each module. we provide three primary contributions regarding faithfulness in nmns. first, we propose the concept of module-wise faithfulness -a systematic evaluation of individual module performance in nmns that judges whether they have learned their intended operations, and define metrics to quantify this for both visual and textual reasoning ( §3). empirically, we show on both nlvr2 @xcite and drop @xcite that training a nmn using end-task supervision, even using gold programs, does not yield modulewise faithfulness, i.e., the modules do not perform their intended reasoning task. second, we provide strategies for improving module-wise faithfulness in nmns ( §4). specifically, (a) we demonstrate how module architecture affects faithfulness ( §4.1), (b) propose supervising module outputs with either a proxy task or heuristically generated data ( §4.2), and (c) show that providing modules with uncontexualized token representations improves faithfulness ( §4.3). figure 1 shows an example where our approach (faithful-nmn) results in expected module outputs as compared to the basic-nmn. last, we collect human-annotated intermediate outputs for 536 examples in nlvr2 and for 215 examples in drop to measure the module-wise faithfulness of models, and publicly release them for future work. our code and data are available at 2 neural module networks overview neural module networks (nmns; @xcite are a class of models that map a natural language utterance into an executable program, composed of learnable modules that can be executed against a given context (images, text, etc.), to produce the utterance's denotation (truth value in nlvr2, or a text answer in drop). modules are designed to solve atomic reasoning tasks and can be composed to perform complex reasoning. for example, in figure 1 , the utterance "all the dogs are black" is mapped to the program equal(count(find[dogs]), count(filter[black](find[dogs]))). the find module is expected to find all dogs in the image and the filter module is expected to output only the black ones from its input. figure 2 shows two other example programs with the expected output of each module in the program. a nmn has two main components: (1) parser, which maps the utterance into an executable program; and (2) executor, which executes the program against the context to produce the denotation. in our setup, programs are always trees where each tree node is a module. in this work, we focus on the executor, and specifically the faithfulness of module execution. we examine nmns for both text and images, and describe their modules next. ## module-wise faithfulness neural module networks (nmns) facilitate interpretability of their predictions via the reasoning steps in the structured program and providing the outputs of those intermediate steps during execution. for example, in figure 2 , all reasoning steps taken by both the visual-nmn and text-nmn can be discerned from the program and the intermediate module outputs. however, because module parameters are learned from an end-task, there is no guarantee that the modules will learn to perform their intended reasoning operation. in such a scenario, when modules do not perform their intended reasoning, the program is no longer a faithful explanation of the model behavior since it is not possible to reliably predict the outputs of the intermediate reasoning steps given the program. work on nmns thus far @xcite has overlooked systematically evaluating faithfulness, performing only qualitative analysis of intermediate outputs. we introduce the concept of module-wise faithfulness aimed at evaluating whether each module has correctly learned its intended operation by judging the correctness of its outputs in a trained nmn. for example, in figure 2 (top), a model would be judged module-wise faithful if the outputs of all the modules, find, relocate, and with relation, are correct -i.e. similar to the outputs that a human would expect. we provide gold programs when evaluating faithfulness, to not conflate faithfulness with parser accuracy. ## improving faithfulness in nmns module-wise faithfulness is affected by various factors: the choice of modules and their implementation ( § 4.1), use of auxiliary supervision ( § 4.2), and the use of contextual utterance embeddings ( § 4.3). we discuss ways of improving faithfulness of nmns across these dimensions. ## experiments we first introduce the datasets used and the experimental setup for measuring faithfulness ( § 5.1). we demonstrate that training nmns using end-task supervision only does not yield module-wise faithfulness both for visual and textual reasoning. we then show that the methods from §4 are crucial for achieving faithfulness and how different design choices affect it ( § 5.2). finally, we qualitatively show examples of improved faithfulness and analyze possible reasons for errors ( § 5.3). ## related work nmns were originally introduced for visual question answering and applied to datasets with syn-utt: "the llamas in both images are eating" 100% 91% 8% 6% a related systematic evaluation of interpretability in vqa was conducted by @xcite . they evaluated the interpretability of their vqa counting model, where the interpretability score is given by the semantic similarity between the gold label for a bounding box and the relevant word(s) in the question. however, they studied only counting questions, which were also far less compositional than those in nlvr2 and drop. similar to the gold module output annotations that we provide and evaluate against, hotpotqa @xcite and coqa @xcite datasets include supporting facts or rationales for the answers to their questions, which can be used for both supervision and evaluation. in concurrent work, jacovi and goldberg (2020) recommend studying faithfulness on a scale rather than as a binary concept. our evaluation method can be viewed as one example of this approach. ## conclusion we introduce the concept of module-wise faithfulness, a systematic evaluation of faithfulness in neural module networks (nmns) for visual and textual reasoning. we show that naïve training of nmns does not produce faithful modules and propose several techniques to improve module-wise faithfulness in nmns. we show how our approach leads to much higher module-wise faithfulness at a low cost to performance. we encourage future work to judge model interpretability using the proposed evaluation and publicly published annotations, and explore techniques for improving faithfulness and interpretability in compositional models.