Model Interpretability and Rationale Extraction by Input Mask Optimization (2023)

Abstract: Concurrent with the rapid progress in neural network-based models in NLP, the need for creating explanations for the predictions of these black-box models has risen steadily. Yet, especially for complex inputs like texts or images, existing interpretability methods still struggle with deriving easily interpretable explanations that also accurately represent the basis for the model’s decision. To this end, we propose a new, model-agnostic method to generate extractive explanations for predictions made by neural networks, that is based on masking parts of the input which the model does not consider to be indicative of the respective class. The masking is done using gradient-based optimization combined with a new regularization scheme that enforces sufficiency, comprehensiveness, and compactness of the generated explanation. Our method achieves state-of-the-art results in a challenging paragraph-level rationale extraction task, showing that this task can be performed without training a specialized model. We further apply our method to image inputs and obtain high-quality explanations for image classifications, which indicates that the objectives for optimizing explanation masks in text generalize to inputs of other modalities.

## introduction black-box machine-learning models like transformers @xcite or convolutional neural networks @xcite are state-of-the-art in natural language processing and computer vision. their complexity enables them to perform well on a variety of tasks, but this comes at the cost of a lack of interpretability: the question of why a model made a specific prediction cannot be answered reliably. especially if such black-box models are used in critical real-world applications (e.g., in the medical domain), this creates a demand for methods that explain network predictions while fulfilling a variety of requirements, like being easy to implement, model and task agnostic, faithful to the inner workings of the network, and producing results that are easily interpretable for humans. to this end, a variety of interpretability methods have been proposed @xcite , but as the aforementioned requirements are often at odds, at least one of them often remains unfulfilled. reasons for this include the reliance on complex message passing schemes that require laborious implementations (e.g., @xcite , the applicability only to specific model architectures (e.g., @xcite , or the fact that explanations often highlight individual, disconnected input features (e.g., standard gradient-based saliency), which contradicts human intuition of a sensible explanation (compare section 2.1 for details). as an example, in a text classification setting, interpretability methods often highlight individual words that explain the prediction, but do not include their context @xcite , even though the context of a word is crucial in determining its meaning: the word "good" influences the prediction in a completely different way if it is preceded by the word "not", meaning that this context has an impact on the classification and should therefore be part of the rationale. notably, this holds true even in the absence of such modifiers, since the context must be available to confirm this absence. in this work, we propose a new method for model explainability that is able to identify parts of the input that are, on the one hand, most indicative of a class and, on the other hand, perceived as a sensible rationale by humans. our method is applicable to all input types that define a spatial structure between individual features (e.g., texts, images) and builds on the assumption that interpretable explanations correspond to smooth and connected regions of features with respect to this spatial structure. it uses numerical optimization to mask out parts of the input that the model does not consider indicative of the class of interest, thus leaving only the parts of the input that are indicative of this class. the masking is done using gradient-based optimization combined with a new regularization scheme that enforces sufficiency, comprehensiveness, and compactness of the generated explanation @xcite , three criteria that have been established in the domain of rationale extraction but are less common in network interpretability methods. in this way, our method bridges the gap between model interpretability and rationale extraction, thereby showing that the latter of which can be performed without training a specialized model, only on the basis of a trained classifier. ## background methods that explain the predictions made by black-box models to users can be broadly categorized into (i) interpretability methods that aim at creating explanations for existing classifiers after they have been trained (section 2.1) and (ii) rationale extraction approaches that are designed to create a rationale as a model output in addition to the usual label prediction (section 2.2). our interpretability method relies on gradient-based input optimization, discussed in detail in section 2.3. ## marc in this section, we introduce marc, our framework for mask-based rationale creation. section 3.1 develops the general framework. sections 3.2 and 3.3 address the specificities of applying marc to texts and images, respectively. ## experiments on rationale extraction we evaluate marc on rationale extraction, a task that is concerned with predicting the correct label for a given textual input while also providing a subset of the input as a rationale for the prediction. ## experiments on image classification we evaluate marc on the task of creating rationales for classifications of imagenet @xcite images. a visual comparison of masks created by marc and other interpretability approaches for resnet-101 @xcite and the vision transformer vit-b/16 @xcite is displayed in figures 2 and 3 , respectively (see appendix c for with more visualizations). we see that marc is able to produce sharp masks that often cover the complete object of interest in the image. for vit-b/16, we include a visualization highlighting the distinction between class-discriminative vs. class-indicative information (compare section 3.1): figure 3b ) used the softmax of the model output as scoring function, which leads marc to highlight only the head and tail of the animal, the two parts that the model uses to differentiate the correct class from the other classes. for figure 3c ), on the other hand, a mixture of the sigmoid of the class logit and the softmax of the model output was used with a ratio of 9:1, making the model highlight all parts in the image that indicate the ground-truth class, as long as they are not significant indicators of other classes. we also evaluate the faithfulness of the explanation created by marc and a variety of other interpretability methods using the same metrics as for textual inputs. for this experiment, we use pretrained resnet-101 and vit-b/16 models on a random sample of 500 imagenet validation images, with further implementational details being described in appendix a. as shown in ## conclusion we propose a new method for creating explanations for neural network predictions that are faithful to the model's reasoning process as well as being sensible with respect to human judgment. we achieve state-of-the-art results on the task of rationale extraction, achieve competitive or state-of-the-art results with respect to faithfulness, and provide visually sensible explanations for classifications of images. as marc is model-agnostic, we believe it to be a useful tool in many areas of machine learning that include textual or image inputs. we further believe that other domains can make use of marc, including multimodal tasks that, for example, combine textual and image inputs, as well as other domains that fulfill the requirements on the spatial structure of the input, like auditory data.