Original: V IS FIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives
Revised: V IS FIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives

Original: Abstract
Revised: Abstract

Original: Many past works aim to improve visual reasoning in models by supervising feature importance (estimated by model explanation techniques) with human annotations such as highlights of important image regions.
Revised: Many past works aim to improve visual reasoning in models by supervising feature importance (estimated by model explanation techniques) with human annotations such as highlights of important image regions.

Original: However, recent work has shown that performance gains from feature importance (FI) supervision for Visual Question Answering (VQA) tasks persist even with random supervision, suggesting that these methods do not meaningfully align model FI with human FI.
Revised: However, recent work has shown that performance gains from feature importance (FI) supervision for Visual Question Answering (VQA) tasks persist even with random supervision, suggesting that these methods do not meaningfully align model FI with human FI.

Original: In this paper, we show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason (RRR) metrics by optimizing for four key model objectives: (1) accurate predictions given limited but sufficient information (Sufficiency); (2) max-entropy predictions given no important information (Uncertainty); (3) invariance of predictions to changes in unimportant features (Invariance); and (4) alignment between model FI explanations and human FI explanations (Plausibility).
Revised: In this paper, we show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason (RRR) metrics by optimizing for four key model objectives: (1) accurate predictions given limited but sufficient information (Sufficiency); (2) max-entropy predictions given no important information (Uncertainty); (3) invariance of predictions to changes in unimportant features (Invariance); and (4) alignment between model FI explanations and human FI explanations (Plausibility).

Original: Our best performing method,
Revised: Our best performing method,

Original: Visual Feature Importance Supervision (V IS FIS), outperforms strong baselines on benchmark VQA datasets in terms of both in-distribution and out-of-distribution accuracy.
Revised: Visual Feature Importance Supervision (V IS FIS), outperforms strong baselines on benchmark VQA datasets in terms of both in-distribution and out-of-distribution accuracy.

Original: While past work suggests that the mechanism for improved accuracy is through improved explanation plausibility, we show that this relationship depends crucially on explanation faithfulness (whether explanations truly represent the model’s internal reasoning).
Revised: While past work suggests that the mechanism for improved accuracy is through improved explanation plausibility, we show that this relationship depends crucially on explanation faithfulness (whether explanations truly represent the model’s internal reasoning).

Original: Predictions are more accurate when explanations are plausible and faithful, and not when they are plausible but not faithful.
Revised: Predictions are more accurate when explanations are plausible and faithful, and not when they are plausible but not faithful.

Original: Lastly, we show that, surprisingly, RRR metrics are not predictive of out-of-distribution model accuracy when controlling for a model’s in-distribution accuracy, which calls into question the value of these metrics for evaluating model reasoning.
Revised: Lastly, we show that, surprisingly, RRR metrics are not predictive of out-of-distribution model accuracy when controlling for a model’s in-distribution accuracy, which calls into question the value of these metrics for evaluating model reasoning.

Original:  Introduction
Revised:  Introduction

Original: Many past works aim to teach models to ignore spurious features by making use of additional information about which features in an input are important [31, 46, 61].
Revised: Many past works aim to teach models to ignore spurious features by making use of additional information about which features in an input are important [31, 46, 61].

Original: For example, individual words can be annotated as (un)important in NLP tasks [46, 66], or regions of image pixels can be highlighted by humans as extra supervision for vision tasks [11, 51].
Revised: For example, individual words can be annotated as (un)important in NLP tasks [46, 66], or regions of image pixels can be highlighted by humans as extra supervision for vision tasks [11, 51].

Original: In this broad class of feature importance (FI) supervision methods, human annotations of important features are typically provided for individual datapoints, and methods often use data augmentation or gradient supervision to encourage models to rely only on important features when making predictions.
Revised: In this broad class of feature importance (FI) supervision methods, human annotations of important features are typically provided for individual datapoints, and methods often use data augmentation or gradient supervision to encourage models to rely only on important features when making predictions.

Original: Such approaches have seen performance improvements in image classification [50, 7], text classification [35, 66, 58], and multimodal visual question answering (VQA) tasks [47, 64, 36].
Revised: Such approaches have seen performance improvements in image classification [50, 7], text classification [35, 66, 58], and multimodal visual question answering (VQA) tasks [47, 64, 36].

Original: [Figure]
Revised: [Figure]

Original: One of the primary motivations behind these approaches is to improve task accuracy by making models “Right for the Right Reasons” [46].
Revised: One of the primary motivations behind these approaches is to improve task accuracy by making models “Right for the Right Reasons” [46].

Original: While existing FI supervision methods for VQA models can improve accuracy [47, 64], recent work has cast doubt on whether human supervision is the source of these improvements.
Revised: While existing FI supervision methods for VQA models can improve accuracy [47, 64], recent work has cast doubt on whether human supervision is the source of these improvements.

Original: Specifically, Shrestha et al.
Revised: Specifically, Shrestha et al.

Original: [48] show that VQA improvements persist even when random image FI annotations are used as supervision, suggesting that existing approaches may not extract meaningful signal from the human annotations.
Revised: [48] show that VQA improvements persist even when random image FI annotations are used as supervision, suggesting that existing approaches may not extract meaningful signal from the human annotations.

Original: Motivated by this shortcoming, we explore several aspects of the FI supervision problem in the context of VQA tasks:
Revised: Motivated by this shortcoming, we explore several aspects of the FI supervision problem in the context of VQA tasks:

Original: Improving VQA accuracy with FI supervision via four key model objectives (Sec. 7.1).
Revised: Improving VQA accuracy with FI supervision via four key model objectives (Sec. 7.1).

Original: Past
Revised: Past

Original: VQA methods focus on data augmentation techniques [36] or ways to directly supervise model feature importance [47, 64].
Revised: VQA methods focus on data augmentation techniques [36] or ways to directly supervise model feature importance [47, 64].

Original: We make use of four key objectives (represented in Fig.
Revised: We make use of four key objectives (represented in Fig.

Original: 1): (1) a Sufficiency objective encouraging the model to predict the correct label given only important input features [7]; (2) an Uncertainty objective encouraging max-entropy outputs when given only unimportant features; (3) an Invariance objective encouraging model outputs to be invariant to changes in unimportant features [46]; and (4) an Align objective that penalizes the model when its FI estimates differ from human
Revised: 1): (1) a Sufficiency objective encouraging the model to predict the correct label given only important input features [7]; (2) an Uncertainty objective encouraging max-entropy outputs when given only unimportant features; (3) an Invariance objective encouraging model outputs to be invariant to changes in unimportant features [46]; and (4) an Align objective that penalizes the model when its FI estimates differ from human

Original: FI annotations [47].
Revised: FI annotations [47].

Original: Our best performing method, termed Visual Feature Importance Supervision (V IS FIS), combines these strategies strategies in a novel manner, improving both in-distribution and out-of-distribution accuracy.
Revised: Our best performing method, termed Visual Feature Importance Supervision (V IS FIS), combines these strategies strategies in a novel manner, improving both in-distribution and out-of-distribution accuracy.

Original: Following guidelines from Shrestha et al.
Revised: Following guidelines from Shrestha et al.

Original: [48], we show that this improvement does not occur with random supervision, meaning V IS FIS learns from human supervision itself.
Revised: [48], we show that this improvement does not occur with random supervision, meaning V IS FIS learns from human supervision itself.

Original: Lastly, after analyzing how explanation plausibility and faithfulness [23] relate to accuracy at the datapoint level, we suggest that FI supervision improves prediction accuracy by improving the plausibility of faithful FI explanations, rather than plausibility alone as past work has suggested [47, 7].
Revised: Lastly, after analyzing how explanation plausibility and faithfulness [23] relate to accuracy at the datapoint level, we suggest that FI supervision improves prediction accuracy by improving the plausibility of faithful FI explanations, rather than plausibility alone as past work has suggested [47, 7].

Original: Evaluating models on new Right for the Right Reason (RRR) metrics (Sec. 7.4).
Revised: Evaluating models on new Right for the Right Reason (RRR) metrics (Sec. 7.4).

Original: Beyond measuring model accuracy, past works evaluate models on a few Right for the Right Reason metrics in order to understand whether model reasoning is correct [47, 48, 65, 7, 19, 16].
Revised: Beyond measuring model accuracy, past works evaluate models on a few Right for the Right Reason metrics in order to understand whether model reasoning is correct [47, 48, 65, 7, 19, 16].

Original: Correct reasoning is valuable because it suggests that models will generalize to test data that we might not be able to verify their performance on, which can occur e.g. when there are exponentially many cases we wish to test or when such data is prohibitively expensive to collect.
Revised: Correct reasoning is valuable because it suggests that models will generalize to test data that we might not be able to verify their performance on, which can occur e.g. when there are exponentially many cases we wish to test or when such data is prohibitively expensive to collect.

Original: We propose a broad set of RRR metrics for model evaluation, with similar motivation to our key model objectives above (see Fig. 1).
Revised: We propose a broad set of RRR metrics for model evaluation, with similar motivation to our key model objectives above (see Fig. 1).

Original: Specifically, in addition to measuring existing metrics for (1) in-distribution accuracy, (2) out-of-distribution accuracy [1, 10, 6, 8, 7], and (3) model accuracy on sufficient feature subsets (RRR-Suff) [7], we also evaluate (4) model uncertainty given uninformative inputs (RRR-Unc), and (5) model invariance to changes in unimportant features (RRR-Inv).
Revised: Specifically, in addition to measuring existing metrics for (1) in-distribution accuracy, (2) out-of-distribution accuracy [1, 10, 6, 8, 7], and (3) model accuracy on sufficient feature subsets (RRR-Suff) [7], we also evaluate (4) model uncertainty given uninformative inputs (RRR-Unc), and (5) model invariance to changes in unimportant features (RRR-Inv).

Original: These metrics help verify that models can: arrive at correct answers relying only on features that are actually important (metric #3), are invariant to the addition or removal of unimportant features that should not affect the label (metric #4), and are appropriately uncertain about the model class when the input contains
Revised: These metrics help verify that models can: arrive at correct answers relying only on features that are actually important (metric #3), are invariant to the addition or removal of unimportant features that should not affect the label (metric #4), and are appropriately uncertain about the model class when the input contains

Original: Predicting model generalization to OOD data with RRR metrics (Sec. 7.5).
Revised: Predicting model generalization to OOD data with RRR metrics (Sec. 7.5).

Original: The practical value of the above RRR metrics can be considered in terms of their ability to inform us about model performance on data that we are not able to test on.
Revised: The practical value of the above RRR metrics can be considered in terms of their ability to inform us about model performance on data that we are not able to test on.

Original: We simulate this situation by evaluating models on in-distribution (ID) data and predicting whether the models will generalize to OOD data based on their accuracy and RRR metrics for ID data.
Revised: We simulate this situation by evaluating models on in-distribution (ID) data and predicting whether the models will generalize to OOD data based on their accuracy and RRR metrics for ID data.

Original: Surprisingly, we find that both existing RRR metrics and our new ones do not better predict OOD accuracy than ID accuracy does on its own.
Revised: Surprisingly, we find that both existing RRR metrics and our new ones do not better predict OOD accuracy than ID accuracy does on its own.

Original: This finding suggests that these metrics may not be a good evaluation of model reasoning, and that there is no good replacement yet for evaluating model accuracy on OOD data in addition to ID data.
Revised: This finding suggests that these metrics may not be a good evaluation of model reasoning, and that there is no good replacement yet for evaluating model accuracy on OOD data in addition to ID data.

Original: In summary of our contributions, we show that:
Revised: In summary of our contributions, we show that:

Original: FI supervision can improve both ID and OOD model accuracy on several benchmark VQA datasets.
Revised: FI supervision can improve both ID and OOD model accuracy on several benchmark VQA datasets.

Original: In particular, V IS FIS improves over unsupervised baselines and the previous state-of-the-art on CLEVR-XAI by up to 4.7 points on OOD data.
Revised: In particular, V IS FIS improves over unsupervised baselines and the previous state-of-the-art on CLEVR-XAI by up to 4.7 points on OOD data.

Original: Explanation plausibility correlates with model accuracy only when explanations are also faithful, which sheds light on the mechanism by which FI supervision improves model accuracy.
Revised: Explanation plausibility correlates with model accuracy only when explanations are also faithful, which sheds light on the mechanism by which FI supervision improves model accuracy.

Original: FI supervision improves model performance on several RRR metrics, including new invariance and uncertainty metrics.
Revised: FI supervision improves model performance on several RRR metrics, including new invariance and uncertainty metrics.

Original: RRR metrics do not correlate better with OOD accuracy than ID model accuracy does on its own.
Revised: RRR metrics do not correlate better with OOD accuracy than ID model accuracy does on its own.

Original: Consequently, RRR metrics may not be as valuable as previously thought.
Revised: Consequently, RRR metrics may not be as valuable as previously thought.

Original:  Related Work
Revised:  Related Work

Original: Supervising FI explanations.
Revised: Supervising FI explanations.

Original: Past works primarily supervise gradient-based [46] or attention-based model explanations [66, 52, 17, 9].
Revised: Past works primarily supervise gradient-based [46] or attention-based model explanations [66, 52, 17, 9].

Original: For example, Ross et al.
Revised: For example, Ross et al.

Original: [46] enforce an ℓ 2 norm on the gradient of the loss w.r.t.
Revised: [46] enforce an ℓ 2 norm on the gradient of the loss w.r.t.

Original: the model input for features marked as unimportant by a human FI explanation.
Revised: the model input for features marked as unimportant by a human FI explanation.

Original: This method appears in several later works [50, 18, 51].
Revised: This method appears in several later works [50, 18, 51].

Original: In a VQA setting, Selvaraju et al.
Revised: In a VQA setting, Selvaraju et al.

Original: [47] and Wu and Mooney [64] align the entire input gradient with human FI.
Revised: [47] and Wu and Mooney [64] align the entire input gradient with human FI.

Original: In addition to using input gradients (termed Vanilla Gradient), we consider the Expected Gradients method [15], a computationally efficient implementation of Integrated Gradients [54].
Revised: In addition to using input gradients (termed Vanilla Gradient), we consider the Expected Gradients method [15], a computationally efficient implementation of Integrated Gradients [54].

Original: Omission or perturbation-based approaches have seen more limited use.
Revised: Omission or perturbation-based approaches have seen more limited use.

Original: Kennedy et al.
Revised: Kennedy et al.

Original: [28] regularize omission-based FI toward 0 for group identifiers in hate speech detection.
Revised: [28] regularize omission-based FI toward 0 for group identifiers in hate speech detection.

Original: In addition to simple leave-one-out [33] and keep-one-in methods, we propose a differentiable version of the popular linear method, SHAP [37].
Revised: In addition to simple leave-one-out [33] and keep-one-in methods, we propose a differentiable version of the popular linear method, SHAP [37].

Original: For a survey of methods we refer readers to Friedrich et al.
Revised: For a survey of methods we refer readers to Friedrich et al.

Original: Following the analysis of Shrestha et al.
Revised: Following the analysis of Shrestha et al.

Original: [48], we use a random supervision baseline to show that V IS FIS succeeds by virtue of additional supervision and not simply via model regularization.
Revised: [48], we use a random supervision baseline to show that V IS FIS succeeds by virtue of additional supervision and not simply via model regularization.

Original: Supervised data augmentation.
Revised: Supervised data augmentation.

Original: This line of work uses human explanations to guide data augmentation, sometimes in a human-in-the-loop manner.
Revised: This line of work uses human explanations to guide data augmentation, sometimes in a human-in-the-loop manner.

Original: For instance, Teney et al.
Revised: For instance, Teney et al.

Original: [58] present LIME explanations to people and solicit feedback that is converted into counterexamples for model training.
Revised: [58] present LIME explanations to people and solicit feedback that is converted into counterexamples for model training.

Original: Liang et al.
Revised: Liang et al.

Original: [34] use expert natural language counterfactual explanations to manufacture new labeled inputs.
Revised: [34] use expert natural language counterfactual explanations to manufacture new labeled inputs.

Original: We build on prior work for our data augmentation objectives.
Revised: We build on prior work for our data augmentation objectives.

Original: Similar to Chang et al.
Revised: Similar to Chang et al.

Original: [7] and concurrent work from Singla et al.
Revised: [7] and concurrent work from Singla et al.

Original: [51], our Sufficiency objective encourages models to be accurate given inputs with sufficient features selected according to human FI.
Revised: [51], our Sufficiency objective encourages models to be accurate given inputs with sufficient features selected according to human FI.

Original: Our Uncertainty objective reduces model confidence when no important features are provided, while the most related objectives from Chang et al.
Revised: Our Uncertainty objective reduces model confidence when no important features are provided, while the most related objectives from Chang et al.

Original: [7] and Liu et al.
Revised: [7] and Liu et al.

Original: [36] encourage a different answer rather than an uncertain output.
Revised: [36] encourage a different answer rather than an uncertain output.

Original: We are not aware of objectives encouraging invariance to changes in unimportant features as our invariance objective does.
Revised: We are not aware of objectives encouraging invariance to changes in unimportant features as our invariance objective does.

Original: Other concurrent work encourages models to always predict the true label even when unimportant features are swapped with other unimportant features from the data distribution [19].
Revised: Other concurrent work encourages models to always predict the true label even when unimportant features are swapped with other unimportant features from the data distribution [19].

Original: Right for the Right Reason metrics.
Revised: Right for the Right Reason metrics.

Original: Only a few past works explicitly evaluate RRR metrics in addition to test set accuracy.
Revised: Only a few past works explicitly evaluate RRR metrics in addition to test set accuracy.

Original: Our metrics include the existing RRR-Suff metric [7] and our RRR-Inv and RRR-Unc metrics.
Revised: Our metrics include the existing RRR-Suff metric [7] and our RRR-Inv and RRR-Unc metrics.

Original: While explanation plausibility is regularly proposed as an RRR metric [47, 48, 65, 43, 16], we show that the relationship between accuracy and plausibility is controlled by explanation faithfulness, meaning that plausibility on its own should not be an RRR metric.
Revised: While explanation plausibility is regularly proposed as an RRR metric [47, 48, 65, 43, 16], we show that the relationship between accuracy and plausibility is controlled by explanation faithfulness, meaning that plausibility on its own should not be an RRR metric.

Original: Recently,
Revised: Recently,

Original: Joshi et al.
Revised: Joshi et al.

Original: [27] propose a number of distinct distribution shifts in text classification for evaluating FI supervision techniques according to model OOD accuracy.
Revised: [27] propose a number of distinct distribution shifts in text classification for evaluating FI supervision techniques according to model OOD accuracy.

Original: We use a “changing prior” distribution shift standardly used in past work for VQA [11, 59], and moreover, the focus of our work is on novel
Revised: We use a “changing prior” distribution shift standardly used in past work for VQA [11, 59], and moreover, the focus of our work is on novel

Original: RRR objectives, metrics, and analysis of how supervision improves models.
Revised: RRR objectives, metrics, and analysis of how supervision improves models.

Original: New machine learning metrics are often justified from first principles or on the basis of a strong correlation with some other good metric, like a human rating [41].
Revised: New machine learning metrics are often justified from first principles or on the basis of a strong correlation with some other good metric, like a human rating [41].

Original: Here, we assess RRR metrics on the basis of their correlation with OOD accuracy, while controlling for model ID accuracy.
Revised: Here, we assess RRR metrics on the basis of their correlation with OOD accuracy, while controlling for model ID accuracy.

Original: This means that we measure the correlation between ID and OOD accuracy like in past studies [57, 39, 38, 60], but we also consider RRR metrics as additional explanatory variables for OOD generalization.
Revised: This means that we measure the correlation between ID and OOD accuracy like in past studies [57, 39, 38, 60], but we also consider RRR metrics as additional explanatory variables for OOD generalization.

Original:  Terminology and Notation
Revised:  Terminology and Notation

Original: FI Explanations.
Revised: FI Explanations.

Original: We distinguish between a human explanation e and a model explanation ˜
Revised: We distinguish between a human explanation e and a model explanation ˜

Original: e that is obtained algorithmically to explain how a model arrived at its prediction for some datapoint.
Revised: e that is obtained algorithmically to explain how a model arrived at its prediction for some datapoint.

Original: In this paper, human explanations are real-valued annotations for input features (which are bounding box representations for each of our datasets).
Revised: In this paper, human explanations are real-valued annotations for input features (which are bounding box representations for each of our datasets).

Original: The score for each bounding box is an indication of its importance to determining the datapoint label, which could roughly be thought of as an answer to the question, “why did data point x receive label y ”
Revised: The score for each bounding box is an indication of its importance to determining the datapoint label, which could roughly be thought of as an answer to the question, “why did data point x receive label y ”

Original: For several objectives and metrics, we binarize the explanations, selecting a threshold based on the data distribution (see Appendix E).
Revised: For several objectives and metrics, we binarize the explanations, selecting a threshold based on the data distribution (see Appendix E).

Original: Replace Functions.
Revised: Replace Functions.

Original: Both generating and evaluating model explanations often require “hiding” input features from a model.
Revised: Both generating and evaluating model explanations often require “hiding” input features from a model.

Original: In practice, we must replace features with some baseline value [53].
Revised: In practice, we must replace features with some baseline value [53].

Original: One simple and common way to replace features is to use an all-zeros feature [32, 54, 4].
Revised: One simple and common way to replace features is to use an all-zeros feature [32, 54, 4].

Original: We compare among several Replace functions to find the best function for learning from FI supervision (see
Revised: We compare among several Replace functions to find the best function for learning from FI supervision (see

Original: Appendix B).
Revised: Appendix B).

Original: We ultimately select the All-Negative-Ones function, which replaces a bounding box feature vector with the all negative ones vector, {− 1 } d .
Revised: We ultimately select the All-Negative-Ones function, which replaces a bounding box feature vector with the all negative ones vector, {− 1 } d .

Original: We use x e to denote a version of the input x where features where e is 0 are replaced via our Replace function.
Revised: We use x e to denote a version of the input x where features where e is 0 are replaced via our Replace function.

Original: Model Notation.
Revised: Model Notation.

Original: We parametrize a distribution p ( y | x ) =
Revised: We parametrize a distribution p ( y | x ) =

Original: f θ ( x ) for classification purposes.
Revised: f θ ( x ) for classification purposes.

Original: Here, ˆ y = arg max y f θ ( x ) y is the model prediction, f θ ( x ) ˆ y is the predicted probability, and Y is the space of eligible answers, which is a large set that is shared across all questions in our VQA tasks.
Revised: Here, ˆ y = arg max y f θ ( x ) y is the model prediction, f θ ( x ) ˆ y is the predicted probability, and Y is the space of eligible answers, which is a large set that is shared across all questions in our VQA tasks.

Original:  Methods for Learning from Human Feature Importance Supervision
Revised:  Methods for Learning from Human Feature Importance Supervision

Original: We now describe how to optimize for several key model desiderata using human FI supervision (represented in Fig. 1).
Revised: We now describe how to optimize for several key model desiderata using human FI supervision (represented in Fig. 1).

Original: In Sec. 4.5, we give the overall objective for Visual Feature Importance Supervision (V IS FIS), which combines the objective terms below in order to improve model generalization.
Revised: In Sec. 4.5, we give the overall objective for Visual Feature Importance Supervision (V IS FIS), which combines the objective terms below in order to improve model generalization.

Original: 4.1 Accuracy Given Sufficient Information
Revised: 4.1 Accuracy Given Sufficient Information

Original: Goal.
Revised: Goal.

Original: Like Chang et al.
Revised: Like Chang et al.

Original: [7], we hope for image processing models to make accurate predictions given subsets of image features that are sufficient for arriving at the correct label, since this suggests that a model recognizes that the important features are in fact important.
Revised: [7], we hope for image processing models to make accurate predictions given subsets of image features that are sufficient for arriving at the correct label, since this suggests that a model recognizes that the important features are in fact important.

Original: Method.
Revised: Method.

Original: Access to human explanations should enable us to automatically construct sufficient inputs with some amount of unimportant information removed [7].
Revised: Access to human explanations should enable us to automatically construct sufficient inputs with some amount of unimportant information removed [7].

Original: In particular, for every input we can create another datapoint by using the human explanation e to Replace unimportant features, while keeping the same label.
Revised: In particular, for every input we can create another datapoint by using the human explanation e to Replace unimportant features, while keeping the same label.

Original: The corresponding objective is given as:
Revised: The corresponding objective is given as:

Original: [Equation]
Revised: [Equation]

Original: This objective differs from previous instantiations [7] by virtue of the Replace function used (see
Revised: This objective differs from previous instantiations [7] by virtue of the Replace function used (see

Original: Sec. 4.5).
Revised: Sec. 4.5).

Original: We compare L Suff against an unsupervised baseline using random feature subsets.
Revised: We compare L Suff against an unsupervised baseline using random feature subsets.

Original: That is, a random distribution D s specifies how likely it is that we Replace a feature:
Revised: That is, a random distribution D s specifies how likely it is that we Replace a feature:

Original: [Equation]
Revised: [Equation]

Original: which, when training via SGD, is estimated using one sample per datapoint per batch.
Revised: which, when training via SGD, is estimated using one sample per datapoint per batch.

Original: 4.2 Uncertainty Given Only Unimportant Information
Revised: 4.2 Uncertainty Given Only Unimportant Information

Original: Goal.
Revised: Goal.

Original: We would prefer for a model to give uncertain outputs for inputs with no important features, meaning the model should give a near-uniform distribution over classes.
Revised: We would prefer for a model to give uncertain outputs for inputs with no important features, meaning the model should give a near-uniform distribution over classes.

Original: Since there is no evidence for any given class, the model should not be confident the input belongs in a particular class.
Revised: Since there is no evidence for any given class, the model should not be confident the input belongs in a particular class.

Original: Method.
Revised: Method.

Original: With this goal in mind, Chang et al.
Revised: With this goal in mind, Chang et al.

Original: [7] train models to give less confident outputs for images with important foreground features removed.
Revised: [7] train models to give less confident outputs for images with important foreground features removed.

Original: More specifically, they encourage the model to predict any class except the image’s true class.
Revised: More specifically, they encourage the model to predict any class except the image’s true class.

Original: In contrast, we penalize a KL divergence between the model output distribution and a uniform distribution,
Revised: In contrast, we penalize a KL divergence between the model output distribution and a uniform distribution,

Original: [Equation]
Revised: [Equation]

Original: where Unif ( |Y| ) is the uniform distribution and u = 1 − e indicates unimportant features.
Revised: where Unif ( |Y| ) is the uniform distribution and u = 1 − e indicates unimportant features.

Original: 4.3 Invariance to Unimportant Information
Revised: 4.3 Invariance to Unimportant Information

Original: Goal.
Revised: Goal.

Original: We would like models to be invariant to changes in an image’s unimportant features.
Revised: We would like models to be invariant to changes in an image’s unimportant features.

Original: This property is desirable because it means that a model correctly treats unimportant features as unimportant.
Revised: This property is desirable because it means that a model correctly treats unimportant features as unimportant.

Original: Method.
Revised: Method.

Original: We first describe a simple data augmentation approach, then describe an FI supervision approach similar to past work [50, 7].
Revised: We first describe a simple data augmentation approach, then describe an FI supervision approach similar to past work [50, 7].

Original: In a data augmentation approach, we train a model to produce the same outputs for two inputs that share the same important information while differing in the unimportant information they contain.
Revised: In a data augmentation approach, we train a model to produce the same outputs for two inputs that share the same important information while differing in the unimportant information they contain.

Original: Specifically, we use e to obtain an input with both important and unimportant features, denoted by x e ∪ u = Replace( x, e ∪ u ) .
Revised: Specifically, we use e to obtain an input with both important and unimportant features, denoted by x e ∪ u = Replace( x, e ∪ u ) .

Original: Then, we penalize the KL divergence between the output distributions on the two inputs x e and x e ∪ u .
Revised: Then, we penalize the KL divergence between the output distributions on the two inputs x e and x e ∪ u .

Original: The resulting objective is then:
Revised: The resulting objective is then:

Original: [Equation]
Revised: [Equation]

Original: (4) where D u is a distribution over binary vectors.
Revised: (4) where D u is a distribution over binary vectors.

Original: L Inv-DA is estimated with one sample like L Suff-Random .
Revised: L Inv-DA is estimated with one sample like L Suff-Random .

Original: In an FI supervision approach, we first obtain model explanations at the datapoint level as ˜ e =
Revised: In an FI supervision approach, we first obtain model explanations at the datapoint level as ˜ e =

Original: Explain( f θ , x, ˆ y ) , where Explain is a differentiable explanation method (possible methods described below).
Revised: Explain( f θ , x, ˆ y ) , where Explain is a differentiable explanation method (possible methods described below).

Original: Then we seek to directly penalize models for treating unimportant features as important.
Revised: Then we seek to directly penalize models for treating unimportant features as important.

Original: To do so, we encourage FI scores for unimportant features to be 0 :
Revised: To do so, we encourage FI scores for unimportant features to be 0 :

Original: [Equation]
Revised: [Equation]

Original: where ˜ e u is the subset of the explanation over features marked as unimportant by e .
Revised: where ˜ e u is the subset of the explanation over features marked as unimportant by e .

Original: Past work uses an ℓ 2 distance for this objective [50, 7], while we use an ℓ 1 penalty after normalizing explanations to unit length, since explanations from different FI methods have different scales.
Revised: Past work uses an ℓ 2 distance for this objective [50, 7], while we use an ℓ 1 penalty after normalizing explanations to unit length, since explanations from different FI methods have different scales.

Original: We consider a few options for differentiable explanation methods.
Revised: We consider a few options for differentiable explanation methods.

Original: Past work has primarily used gradient-based [46, 47, 50, 18, 64, 7] and attention-based explanations [66, 52, 17, 9].
Revised: Past work has primarily used gradient-based [46, 47, 50, 18, 64, 7] and attention-based explanations [66, 52, 17, 9].

Original: We adopt existing gradient/attention methods and provide new differentiable perturbation-based methods.
Revised: We adopt existing gradient/attention methods and provide new differentiable perturbation-based methods.

Original: Gradient-based explanations.
Revised: Gradient-based explanations.

Original: One can optimize objectives involving gradient-based explanations w.r.t.
Revised: One can optimize objectives involving gradient-based explanations w.r.t.

Original: θ by computing second derivatives like ∇ θ ∇ x f θ ( x ) in a framework like PyTorch [42].
Revised: θ by computing second derivatives like ∇ θ ∇ x f θ ( x ) in a framework like PyTorch [42].

Original: We use a simple Vanilla Gradient method and the Expected Gradients method (see Appendix A).
Revised: We use a simple Vanilla Gradient method and the Expected Gradients method (see Appendix A).

Original: Attention-based explanations.
Revised: Attention-based explanations.

Original: We supervise bounding box attention weights in the UpDn model [2], but early experiments suggest this is not an effective method and we do not explore it further.
Revised: We supervise bounding box attention weights in the UpDn model [2], but early experiments suggest this is not an effective method and we do not explore it further.

Original: Perturbation-based explanations.
Revised: Perturbation-based explanations.

Original: Perturbation-based methods like SHAP [37] are very popular explanation methods, but have seen only limited use for FI supervision [28].
Revised: Perturbation-based methods like SHAP [37] are very popular explanation methods, but have seen only limited use for FI supervision [28].

Original: We consider a leave-oneout method (LOO), a keep-one-in method (KOI), Average Effect, and SHAP (see Appendix A).
Revised: We consider a leave-oneout method (LOO), a keep-one-in method (KOI), Average Effect, and SHAP (see Appendix A).

Original: In Appendix D, we discuss limits on the compute budget used for each method during model training.
Revised: In Appendix D, we discuss limits on the compute budget used for each method during model training.

Original: 4.4 Aligned Model and Human Feature Importance
Revised: 4.4 Aligned Model and Human Feature Importance

Original: Goal.
Revised: Goal.

Original: Alignment between human and model explanations has frequently been proposed as a goal for models [47, 48, 65, 16].
Revised: Alignment between human and model explanations has frequently been proposed as a goal for models [47, 48, 65, 16].

Original: In general, past works assume that model explanations are faithful , meaning they accurately communicate a model’s internal reasoning [23].
Revised: In general, past works assume that model explanations are faithful , meaning they accurately communicate a model’s internal reasoning [23].

Original: This assumption is necessary for the alignment between model and human explanations, termed plausibility by Jacovi and Goldberg [23], to be evidence that model reasoning is similar to human reasoning.
Revised: This assumption is necessary for the alignment between model and human explanations, termed plausibility by Jacovi and Goldberg [23], to be evidence that model reasoning is similar to human reasoning.

Original: Of course, model explanations are not guaranteed to be faithful.
Revised: Of course, model explanations are not guaranteed to be faithful.

Original: To the extent that they are faithful, however, encouraging explanation plausibility during training may help align model reasoning with human reasoning.
Revised: To the extent that they are faithful, however, encouraging explanation plausibility during training may help align model reasoning with human reasoning.

Original: Method.
Revised: Method.

Original: We first obtain model explanations at the datapoint level as ˜ e = Explain( f θ , x, ˆ y )
Revised: We first obtain model explanations at the datapoint level as ˜ e = Explain( f θ , x, ˆ y )

Original: (see Sec.
Revised: (see Sec.

Original: 4.3 above).
Revised: 4.3 above).

Original: Then, we can measure the difference between ˜ e and the human explanation e using an l p distance, cosine similarity, or a differentiable ranking function [47, 64].
Revised: Then, we can measure the difference between ˜ e and the human explanation e using an l p distance, cosine similarity, or a differentiable ranking function [47, 64].

Original: We use a cosine similarity since model explanations and human explanations do not share the same scale.
Revised: We use a cosine similarity since model explanations and human explanations do not share the same scale.

Original: Our objective is thus:
Revised: Our objective is thus:

Original: [Equation]
Revised: [Equation]

Original: 4.5 Overall Objective for V IS FIS: Visual Feature Importance Supervision
Revised: 4.5 Overall Objective for V IS FIS: Visual Feature Importance Supervision

Original: We combine the supervised objective terms from above to achieve the corresponding model desiderata simultaneously.
Revised: We combine the supervised objective terms from above to achieve the corresponding model desiderata simultaneously.

Original: Following objective tuning experiments showing that Inv-FI outperforms Inv-DA (see
Revised: Following objective tuning experiments showing that Inv-FI outperforms Inv-DA (see

Original: Appendix Table 11)
Revised: Appendix Table 11)

Original: , we use Inv-FI rather than Inv-DA, and therefore our final V IS FIS objective is:
Revised: , we use Inv-FI rather than Inv-DA, and therefore our final V IS FIS objective is:

Original: [Equation]
Revised: [Equation]

Original: where L Task is a standard supervised cross-entropy loss.
Revised: where L Task is a standard supervised cross-entropy loss.

Original: Besides tuning the values of λ i one at a time, we also tune the Replace function and FI method used in this objective, making sure to use comparable compute budgets across FI methods.
Revised: Besides tuning the values of λ i one at a time, we also tune the Replace function and FI method used in this objective, making sure to use comparable compute budgets across FI methods.

Original: Replace functions we consider are listed in Appendix B (results in Table 6), and FI methods in Appendix A (results in Tables 9 and 10).
Revised: Replace functions we consider are listed in Appendix B (results in Table 6), and FI methods in Appendix A (results in Tables 9 and 10).

Original: Following tuning, we find that it is preferable to Replace bounding box representations with the negative ones vector, {− 1 } d , and surprisingly, we find that Vanilla Gradient is the best performing
Revised: Following tuning, we find that it is preferable to Replace bounding box representations with the negative ones vector, {− 1 } d , and surprisingly, we find that Vanilla Gradient is the best performing

Original: FI method, surpassing all perturbation-based methods as well as the Expected Gradients method.
Revised: FI method, surpassing all perturbation-based methods as well as the Expected Gradients method.

Original:  Metrics
Revised:  Metrics

Original: Next, we describe the RRR and explanation metrics for each of our model desiderata outlined above.
Revised: Next, we describe the RRR and explanation metrics for each of our model desiderata outlined above.

Original: We also measure model ID and OOD accuracy (distribution shifts described in Sec. 6).
Revised: We also measure model ID and OOD accuracy (distribution shifts described in Sec. 6).

Original: As with the model objectives, we use the All-Negative-Ones Replace function as needed.
Revised: As with the model objectives, we use the All-Negative-Ones Replace function as needed.

Original: RRR-Suffiency.
Revised: RRR-Suffiency.

Original: We measure model accuracy on inputs containing only features selected as important by their respective human explanation (similar to [7]).
Revised: We measure model accuracy on inputs containing only features selected as important by their respective human explanation (similar to [7]).

Original: The remaining features are Replaced .
Revised: The remaining features are Replaced .

Original: RRR-Uncertainty.
Revised: RRR-Uncertainty.

Original: We propose to measure how uncertain the model prediction is given only unimportant information.
Revised: We propose to measure how uncertain the model prediction is given only unimportant information.

Original: Specifically, we report the average model predicted probability when we provide only unimportant features to the model (according to the human explanation), so lower is better.
Revised: Specifically, we report the average model predicted probability when we provide only unimportant features to the model (according to the human explanation), so lower is better.

Original: RRR-Invariance.
Revised: RRR-Invariance.

Original: We propose to calculate the agreement between model predictions with the input x e and three x e ∪ u that each include a random number of unimportant features.
Revised: We propose to calculate the agreement between model predictions with the input x e and three x e ∪ u that each include a random number of unimportant features.

Original: The final metric is averaged over three random u for each test point, then over all test points.
Revised: The final metric is averaged over three random u for each test point, then over all test points.

Original: Explanation Plausibility.
Revised: Explanation Plausibility.

Original: Our explanation plausibility metric is the Spearman’s rank correlation between the human and model feature importance vectors, similar to past work [11, 47].
Revised: Our explanation plausibility metric is the Spearman’s rank correlation between the human and model feature importance vectors, similar to past work [11, 47].

Original: We use continuous FI estimates in order to calculate the rank correlation.
Revised: We use continuous FI estimates in order to calculate the rank correlation.

Original: A rank correlation is preferrable here because human and model FI explanations do not lie in the same space.
Revised: A rank correlation is preferrable here because human and model FI explanations do not lie in the same space.

Original: Explanation Faithfulness.
Revised: Explanation Faithfulness.

Original: We use two standard faithfulness metrics [13].
Revised: We use two standard faithfulness metrics [13].

Original: Sufficiency measures whether keeping important features (according to model explanation ˜ e ) leads the model to retain its confidence in its original prediction:
Revised: Sufficiency measures whether keeping important features (according to model explanation ˜ e ) leads the model to retain its confidence in its original prediction:

Original: Suff ( f θ , x, ˜ e ) =
Revised: Suff ( f θ , x, ˜ e ) =

Original: f θ ( x ) ˆ y − f θ ( x ˜ e ) ˆ y .
Revised: f θ ( x ) ˆ y − f θ ( x ˜ e ) ˆ y .

Original: Comprehensiveness measures whether removing important features from an input leads to a decline in model confidence,
Revised: Comprehensiveness measures whether removing important features from an input leads to a decline in model confidence,

Original: Comp ( f θ , x, ˜ e ) =
Revised: Comp ( f θ , x, ˜ e ) =

Original: f θ ( x ) ˆ y − f θ ( x C ˜ e )
Revised: f θ ( x ) ˆ y − f θ ( x C ˜ e )

Original: ˆ y , where x C ˜ e
Revised: ˆ y , where x C ˜ e

Original: = Replace( x, 1 − ˜ e ) is the complement of features in x ˜ e .
Revised: = Replace( x, 1 − ˜ e ) is the complement of features in x ˜ e .

Original: We average these score over several sparsity levels of ˜ e , keeping or removing the top 10%, 25%, or 50% of features [13].
Revised: We average these score over several sparsity levels of ˜ e , keeping or removing the top 10%, 25%, or 50% of features [13].

Original: Note we compute these metrics using the best available explanation method per dataset, as measured by explanation faithfulness (comparison in Appendix G).
Revised: Note we compute these metrics using the best available explanation method per dataset, as measured by explanation faithfulness (comparison in Appendix G).

Original:  Experiment Setup
Revised:  Experiment Setup

Original: Datasets.
Revised: Datasets.

Original: We perform experiments on three benchmark datasets: CLEVR-XAI [5], GQA [21], and VQA-HAT [11].
Revised: We perform experiments on three benchmark datasets: CLEVR-XAI [5], GQA [21], and VQA-HAT [11].

Original: CLEVR-XAI is an algorithmically generated dataset based on CLEVR [26] and provides ground truth visual segmentation masks for each question.
Revised: CLEVR-XAI is an algorithmically generated dataset based on CLEVR [26] and provides ground truth visual segmentation masks for each question.

Original: CLEVR-XAI is limited in visual varieties and vocabularies, but it offers FI supervision in a controlled, low-noise setting.
Revised: CLEVR-XAI is limited in visual varieties and vocabularies, but it offers FI supervision in a controlled, low-noise setting.

Original: GQA contains compositional reasoning questions over naturalistic images.
Revised: GQA contains compositional reasoning questions over naturalistic images.

Original: GQA also includes the program for generating the questions and the ground-truth scene graph from the Visual Genome dataset [30].
Revised: GQA also includes the program for generating the questions and the ground-truth scene graph from the Visual Genome dataset [30].

Original: This allows us to obtain bounding boxes of relevant objects identified through the question program, which we use as FI supervision.
Revised: This allows us to obtain bounding boxes of relevant objects identified through the question program, which we use as FI supervision.

Original: VQA-HAT is based on VQAv1 [3], including naturalistic images and questions with mouse tracking de-blurring used to collect image FI annotations from humans.
Revised: VQA-HAT is based on VQAv1 [3], including naturalistic images and questions with mouse tracking de-blurring used to collect image FI annotations from humans.

Original: For VQA, we report model performance on the more challenging other type questions as recommended by Teney et al.
Revised: For VQA, we report model performance on the more challenging other type questions as recommended by Teney et al.

Original: [Table]
Revised: [Table]

Original: Distribution Shifts.
Revised: Distribution Shifts.

Original: We create both ID and
Revised: We create both ID and

Original: OOD test sets for each dataset, so we always have four data splits: Train, Dev, Test-ID, and
Revised: OOD test sets for each dataset, so we always have four data splits: Train, Dev, Test-ID, and

Original: Test-OOD (split sizes shown in Table 1).
Revised: Test-OOD (split sizes shown in Table 1).

Original: To obtain OOD data, we use distribution shifts similar to those in VQA-CP, which are intended to vary the linguistic bias between ID and OOD splits [1].
Revised: To obtain OOD data, we use distribution shifts similar to those in VQA-CP, which are intended to vary the linguistic bias between ID and OOD splits [1].

Original: We apply the same procedure for distribution shift on all three datasets for comparability.
Revised: We apply the same procedure for distribution shift on all three datasets for comparability.

Original: In detail, we create groups of questions according to the first few words in each question (indicating the type of question), and allocate groups unevenly into ID and OOD sets, randomly assigning 80% of each group to one set and 20% to the other.
Revised: In detail, we create groups of questions according to the first few words in each question (indicating the type of question), and allocate groups unevenly into ID and OOD sets, randomly assigning 80% of each group to one set and 20% to the other.

Original: The ID set is split into Train, Dev, and Test-ID.
Revised: The ID set is split into Train, Dev, and Test-ID.

Original: Model selection is done according to Dev set performance.
Revised: Model selection is done according to Dev set performance.

Original: We further downsample the very large GQA dataset from to about 100k for training and 20k for other splits.
Revised: We further downsample the very large GQA dataset from to about 100k for training and 20k for other splits.

Original: See Appendix Fig.
Revised: See Appendix Fig.

Original: 5 for training size ablation analysis.
Revised: 5 for training size ablation analysis.

Original: We note that we avoid several pitfalls in evaluating VQA models against distribution shifts, as outlined by Teney et al.
Revised: We note that we avoid several pitfalls in evaluating VQA models against distribution shifts, as outlined by Teney et al.

Original: See Appendix Table 7 for sensitivity analysis with randomly resplit data.
Revised: See Appendix Table 7 for sensitivity analysis with randomly resplit data.

Original: Human Feature Importance.
Revised: Human Feature Importance.

Original: For all datasets, we obtain human FI scores at the bounding box (BB) level for detected BBs from the Faster-RCNN detector [45].
Revised: For all datasets, we obtain human FI scores at the bounding box (BB) level for detected BBs from the Faster-RCNN detector [45].

Original: Following Selvaraju et al.
Revised: Following Selvaraju et al.

Original: [47], for both VQA-HAT and CLEVR-XAI we obtain importance scores from pixel-level annotations as s k
Revised: [47], for both VQA-HAT and CLEVR-XAI we obtain importance scores from pixel-level annotations as s k

Original: = E ki / ( E ki + E ko ) , where s k is the score for the k th detected BB and E ki and E ko are the average pixel-level importance score inside and outside the BB, respectively.
Revised: = E ki / ( E ki + E ko ) , where s k is the score for the k th detected BB and E ki and E ko are the average pixel-level importance score inside and outside the BB, respectively.

Original: VQA-HAT has real-valued pixel-level scores, while for CLEVR-XAI, we set the pixel-level score to 1 for pixels within the segmentation mask and 0 elsewhere.
Revised: VQA-HAT has real-valued pixel-level scores, while for CLEVR-XAI, we set the pixel-level score to 1 for pixels within the segmentation mask and 0 elsewhere.

Original: For GQA, since we have BB level annotations, we calculate the importance score based on the intersection over union (IoU) between ground-truth important BBs and detected BBs: s k = max
Revised: For GQA, since we have BB level annotations, we calculate the importance score based on the intersection over union (IoU) between ground-truth important BBs and detected BBs: s k = max

Original: l ∈G IoU ( B kd , B l gt ) where B kd is the BB of the k th detected object and
Revised: l ∈G IoU ( B kd , B l gt ) where B kd is the BB of the k th detected object and

Original: B l gt is the BB of the l -th ground-truth object.
Revised: B l gt is the BB of the l -th ground-truth object.

Original: With importance scores for each BB, we manually set a threshold for determining important and unimportant objects (0.85, 0.55, and 0.3 for CLEVR-XAI,
Revised: With importance scores for each BB, we manually set a threshold for determining important and unimportant objects (0.85, 0.55, and 0.3 for CLEVR-XAI,

Original: VQA-HAT, and GQA respectively).
Revised: VQA-HAT, and GQA respectively).

Original: See Appendix E for sensitivity analysis for this threshold.
Revised: See Appendix E for sensitivity analysis for this threshold.

Original: Models.
Revised: Models.

Original: We run experiments with UpDn [2] and LXMERT [56].
Revised: We run experiments with UpDn [2] and LXMERT [56].

Original: Both models rely on bounding box representations generated by a pretrained Faster R-CNN model [45] (further details in Appendix F).
Revised: Both models rely on bounding box representations generated by a pretrained Faster R-CNN model [45] (further details in Appendix F).

Original: Hypothesis Testing.
Revised: Hypothesis Testing.

Original: We conduct hypothesis tests via a bootstrap resampling model seeds and datapoints 10k times [14].
Revised: We conduct hypothesis tests via a bootstrap resampling model seeds and datapoints 10k times [14].

Original: We obtain 95% confidence intervals in the same way.
Revised: We obtain 95% confidence intervals in the same way.

Original:  Experiment Results 7.1 Can FI Supervision Improve Model Accuracy for VQA?
Revised:  Experiment Results 7.1 Can FI Supervision Improve Model Accuracy for VQA?

Original: Design.
Revised: Design.

Original: Using UpDn on our three datasets, we compare V IS FIS with previous state-of-the-art FI supervision methods for VQA tasks [47, 64] as well as for image classification [50, 7].
Revised: Using UpDn on our three datasets, we compare V IS FIS with previous state-of-the-art FI supervision methods for VQA tasks [47, 64] as well as for image classification [50, 7].

Original: We give results for LXMERT only on CLEVR-XAI, since GQA and VQA are a part of the pretraining data for LXMERT [56].
Revised: We give results for LXMERT only on CLEVR-XAI, since GQA and VQA are a part of the pretraining data for LXMERT [56].

Original: Note we test on the more challenging other type questions only for VQA, following Teney et al.
Revised: Note we test on the more challenging other type questions only for VQA, following Teney et al.

Original: Selvaraju et al.
Revised: Selvaraju et al.

Original: [47] use L align with a ranking loss to align Vanilla
Revised: [47] use L align with a ranking loss to align Vanilla

Original: Gradient explanations and human FI supervision.
Revised: Gradient explanations and human FI supervision.

Original: Wu and Mooney [64] propose a relaxed version of the ranking loss that binarizes important and unimportant features according to human FI supervision and encourages higher model FI for important objects than unimportant ones.
Revised: Wu and Mooney [64] propose a relaxed version of the ranking loss that binarizes important and unimportant features according to human FI supervision and encourages higher model FI for important objects than unimportant ones.

Original: The other methods we consider all use an L FI-Inv objective with an l 2 penalty on Vanilla Gradient explanations.
Revised: The other methods we consider all use an L FI-Inv objective with an l 2 penalty on Vanilla Gradient explanations.

Original: On top of this, Chang et al.
Revised: On top of this, Chang et al.

Original: [7] add an L Suff objective with a Shuffle Replace function that randomly permutes features rather than replacing them, to preserve the marginal data distribution, and Singla et al.
Revised: [7] add an L Suff objective with a Shuffle Replace function that randomly permutes features rather than replacing them, to preserve the marginal data distribution, and Singla et al.

Original: [51] add an L Suff objective with a Gaussian noise Replace function.
Revised: [51] add an L Suff objective with a Gaussian noise Replace function.

Original: Our unsupervised baselines are models trained with only label supervision or using L Suff-Random .
Revised: Our unsupervised baselines are models trained with only label supervision or using L Suff-Random .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Results.
Revised: Results.

Original: We show results for UpDn in Table 2 and for LXMERT in Table 3.
Revised: We show results for UpDn in Table 2 and for LXMERT in Table 3.

Original: First, we find that FI supervision can meaningfully improve model accuracy.
Revised: First, we find that FI supervision can meaningfully improve model accuracy.

Original: With UpDn on CLEVR-XAI,
Revised: With UpDn on CLEVR-XAI,

Original: V IS FIS improves ID accuracy by 1.1 points (±0.5; p = 1e-4) and OOD accuracy by 4.7 points (±1.4; p< 1e-4) over the strongest baseline without supervision, Suff-Random
Revised: V IS FIS improves ID accuracy by 1.1 points (±0.5; p = 1e-4) and OOD accuracy by 4.7 points (±1.4; p< 1e-4) over the strongest baseline without supervision, Suff-Random

Original: (see Appendix Fig.
Revised: (see Appendix Fig.

Original: 9 for breakdown in improvements by CLEVR question type).
Revised: 9 for breakdown in improvements by CLEVR question type).

Original: Trends are similarly positive on the other datasets and with LXMERT, where
Revised: Trends are similarly positive on the other datasets and with LXMERT, where

Original: V IS FIS outperforms the baseline by 0.48 points (±0.35; p< .01) on ID data and 1.07 points ( p< 1e-4) on OOD data (for results on all VQA question types, see Appendix Table 15).
Revised: V IS FIS outperforms the baseline by 0.48 points (±0.35; p< .01) on ID data and 1.07 points ( p< 1e-4) on OOD data (for results on all VQA question types, see Appendix Table 15).

Original: These improvements do not persist when using random explanations (last row), meaning they are caused by the human supervision.
Revised: These improvements do not persist when using random explanations (last row), meaning they are caused by the human supervision.

Original: Finally, we observe that V IS FIS is the best overall method across datasets and architectures, as other methods typically do not improve accuracy over an unsupervised baseline.
Revised: Finally, we observe that V IS FIS is the best overall method across datasets and architectures, as other methods typically do not improve accuracy over an unsupervised baseline.

Original: The next best method is that of Singla et al.
Revised: The next best method is that of Singla et al.

Original: [51], which improves over an unsupervised baseline only for the GQA dataset with UpDn, but V IS FIS still outperforms Singla et al.
Revised: [51], which improves over an unsupervised baseline only for the GQA dataset with UpDn, but V IS FIS still outperforms Singla et al.

Original: [51] there by 2.
Revised: [51] there by 2.

Original: (±0.55; p< 1e-4) points on ID data and 2.31 points (±0.66; p< 1e-4) on OOD data.
Revised: (±0.55; p< 1e-4) points on ID data and 2.31 points (±0.66; p< 1e-4) on OOD data.

Original: 7.2 How Does FI Supervision Improve Accuracy?
Revised: 7.2 How Does FI Supervision Improve Accuracy?

Original: [Figure]
Revised: [Figure]

Original: Trendlines are logistic regressions.
Revised: Trendlines are logistic regressions.

Original: When explanations are more faithful, their alignment with human explanations better correlates with model accuracy.
Revised: When explanations are more faithful, their alignment with human explanations better correlates with model accuracy.

Original: Design.
Revised: Design.

Original: Past work hypothesizes that FI supervision improves accuracy by aligning model and human FI [47, 7].
Revised: Past work hypothesizes that FI supervision improves accuracy by aligning model and human FI [47, 7].

Original: Surprisingly, we find that the relationship between model test accuracy and average explanation plausibility is fairly weak (linear correlation on UpDn+CLEVR-XAI models is ρ = 0.14±0.19).
Revised: Surprisingly, we find that the relationship between model test accuracy and average explanation plausibility is fairly weak (linear correlation on UpDn+CLEVR-XAI models is ρ = 0.14±0.19).

Original: Here, we argue that plausible explanations alone are not evidence of correct model reasoning, but plausible and faithful explanations are.
Revised: Here, we argue that plausible explanations alone are not evidence of correct model reasoning, but plausible and faithful explanations are.

Original: Using 4 million ID/OOD test predictions from UpDn+CLEVR-XAI models, we visualize trendlines from logistic regressions predicting model accuracy based on plausibility and faithfulness at the datapoint level, grouped into Worst, Middle, and Best faithfulness categories based on Sufficiency/Comprehensiveness metrics (see Appendix E).
Revised: Using 4 million ID/OOD test predictions from UpDn+CLEVR-XAI models, we visualize trendlines from logistic regressions predicting model accuracy based on plausibility and faithfulness at the datapoint level, grouped into Worst, Middle, and Best faithfulness categories based on Sufficiency/Comprehensiveness metrics (see Appendix E).

Original: Results.
Revised: Results.

Original: Fig.
Revised: Fig.

Original: 2 shows that as an explanation for a datapoint becomes more plausible, the model is more likely to correctly predict that point’s label, but only when the explanation is also faithful .
Revised: 2 shows that as an explanation for a datapoint becomes more plausible, the model is more likely to correctly predict that point’s label, but only when the explanation is also faithful .

Original: Indeed, a maximally plausible and faithful explanation has about a 90% chance of being correct,
Revised: Indeed, a maximally plausible and faithful explanation has about a 90% chance of being correct,

Original: [Figure]
Revised: [Figure]

Original: while a minimally plausible but highly faithful explanation has closer to a 12.5% of being correct.
Revised: while a minimally plausible but highly faithful explanation has closer to a 12.5% of being correct.

Original: For unfaithful explanations, plausibility has essentially no relationship with accuracy.
Revised: For unfaithful explanations, plausibility has essentially no relationship with accuracy.

Original: Though these trends are not necessarily causal, they are consistent with the view that when model predictions are correct, it is because their true reasoning (as revealed by faithful explanations) aligns with human reasoning.
Revised: Though these trends are not necessarily causal, they are consistent with the view that when model predictions are correct, it is because their true reasoning (as revealed by faithful explanations) aligns with human reasoning.

Original: Fig.
Revised: Fig.

Original: 3 qualitatively illustrates this relationship among faithfulness, plausibility, and accuracy with example data points and model predictions.
Revised: 3 qualitatively illustrates this relationship among faithfulness, plausibility, and accuracy with example data points and model predictions.

Original: We emphasize that while past work has treated plausibility as an RRR metric [47, 48, 65, 43, 16], the results here demonstrate that plausibility alone cannot be a measure of model correctness.
Revised: We emphasize that while past work has treated plausibility as an RRR metric [47, 48, 65, 43, 16], the results here demonstrate that plausibility alone cannot be a measure of model correctness.

Original: 7.3 Which FI Supervision Objectives Improve Accuracy?
Revised: 7.3 Which FI Supervision Objectives Improve Accuracy?

Original: Design.
Revised: Design.

Original: We ablate across objective terms from Sec.
Revised: We ablate across objective terms from Sec.

Original: 4 for UpDn on CLEVR-XAI.
Revised: 4 for UpDn on CLEVR-XAI.

Original: The weight for each objective term is tuned while using only that objective, then kept fixed when objectives are combined (further details in Appendix F).
Revised: The weight for each objective term is tuned while using only that objective, then kept fixed when objectives are combined (further details in Appendix F).

Original: We consider another kind of ablation experiment where we use random supervision for one objective at a time in V IS FIS, with results in Appendix Table 12.
Revised: We consider another kind of ablation experiment where we use random supervision for one objective at a time in V IS FIS, with results in Appendix Table 12.

Original: Results.
Revised: Results.

Original: In Table 4, we find that each individual objective is valuable on its own, and they do well when combined.
Revised: In Table 4, we find that each individual objective is valuable on its own, and they do well when combined.

Original: Relative to the Baseline OOD accuracy, Suff-Human adds 4.1 points, Unc adds 1.
Revised: Relative to the Baseline OOD accuracy, Suff-Human adds 4.1 points, Unc adds 1.

Original: points, Inv-FI adds 2.08 points, and Align adds 4.81 points.
Revised: points, Inv-FI adds 2.08 points, and Align adds 4.81 points.

Original: When the four objectives are combined in V IS FIS, the improvement rises to 6.98 points.
Revised: When the four objectives are combined in V IS FIS, the improvement rises to 6.98 points.

Original: 7.4 Can FI Supervision Make Models Right for the Right Reasons?
Revised: 7.4 Can FI Supervision Make Models Right for the Right Reasons?

Original: [Table]
Revised: [Table]

Original: Design.
Revised: Design.

Original: We report RRR metrics as well as explanation plausibility for the UpDn+CLEVR-XAI models from our objective ablation above.
Revised: We report RRR metrics as well as explanation plausibility for the UpDn+CLEVR-XAI models from our objective ablation above.

Original: Results.
Revised: Results.

Original: In Table 4, we find that
Revised: In Table 4, we find that

Original: FI supervision generally improves
Revised: FI supervision generally improves

Original: RRR metric scores.
Revised: RRR metric scores.

Original: Compared to the Baseline, V IS FIS achieves 27.8 points better Sufficiency, 13.
Revised: Compared to the Baseline, V IS FIS achieves 27.8 points better Sufficiency, 13.

Original: points better Invariance, and 11.
Revised: points better Invariance, and 11.

Original: points better Uncertainty.
Revised: points better Uncertainty.

Original: The best unsupervised method closes the gap slightly on RRR-Suff and RRR-Inv.
Revised: The best unsupervised method closes the gap slightly on RRR-Suff and RRR-Inv.

Original: Specifically, Suff-Random is only 3.97 points worse than Suff-Human on RRR-Suff, and only 0.53 points worse than Inv-DA on
Revised: Specifically, Suff-Random is only 3.97 points worse than Suff-Human on RRR-Suff, and only 0.53 points worse than Inv-DA on

Original: RRR-Inv.
Revised: RRR-Inv.

Original: It suggests that FI supervision noticeably improves RRR metrics, meaning model behavior better fulfills the theoretical desiderata from Sec. 4.
Revised: It suggests that FI supervision noticeably improves RRR metrics, meaning model behavior better fulfills the theoretical desiderata from Sec. 4.

Original: 7.5 Do RRR Metrics Predict OOD Generalization?
Revised: 7.5 Do RRR Metrics Predict OOD Generalization?

Original: Design.
Revised: Design.

Original: We measure the correlation between RRR metrics (calculated with ID data) and OOD accuracy across a large set of models.
Revised: We measure the correlation between RRR metrics (calculated with ID data) and OOD accuracy across a large set of models.

Original: We report results here for all UpDn models on CLEVR-XAI, with similar results for GQA/VQA and LXMERT given in Appendix Table 16.
Revised: We report results here for all UpDn models on CLEVR-XAI, with similar results for GQA/VQA and LXMERT given in Appendix Table 16.

Original: We consider a few possible model metrics, including several composite metrics that combine model-level metrics.
Revised: We consider a few possible model metrics, including several composite metrics that combine model-level metrics.

Original: To optimally weight the individual metrics, we fit statistical models to the data that predict OOD accuracy given the available metrics.
Revised: To optimally weight the individual metrics, we fit statistical models to the data that predict OOD accuracy given the available metrics.

Original: Since this risks overfitting the composite metrics to the data we have, we perform a cross-validation resampling model-level statistics 10k times, using 90 models’ metrics as training data and 10 for testing each time.
Revised: Since this risks overfitting the composite metrics to the data we have, we perform a cross-validation resampling model-level statistics 10k times, using 90 models’ metrics as training data and 10 for testing each time.

Original: The final metrics we consider are: (1) ID accuracy on its own as a baseline, (2) RRR metrics on their own, (3) ID accuracy plus average model confidence, (4) ID accuracy plus explanation metrics (for plausibility and faithfulness), (5)
Revised: The final metrics we consider are: (1) ID accuracy on its own as a baseline, (2) RRR metrics on their own, (3) ID accuracy plus average model confidence, (4) ID accuracy plus explanation metrics (for plausibility and faithfulness), (5)

Original: ID accuracy plus RRR metrics
Revised: ID accuracy plus RRR metrics

Original: [Table]
Revised: [Table]

Original: and (6) All Metrics, which uses all available metrics.
Revised: and (6) All Metrics, which uses all available metrics.

Original: [Table]
Revised: [Table]

Original: Results.
Revised: Results.

Original: In Table 5, we show the average correlations between each metric and model OOD accuracy achieved in our cross-validation.
Revised: In Table 5, we show the average correlations between each metric and model OOD accuracy achieved in our cross-validation.

Original: Interestingly, we find that RRR metrics do not achieve a better correlation with OOD accuracy than ID accuracy does on its own .
Revised: Interestingly, we find that RRR metrics do not achieve a better correlation with OOD accuracy than ID accuracy does on its own .

Original: ID accuracy alone has a correlation of 0.863 with OOD accuracy, while using ID Acc +
Revised: ID accuracy alone has a correlation of 0.863 with OOD accuracy, while using ID Acc +

Original: RRR metrics achieves a correlation of 0.852.
Revised: RRR metrics achieves a correlation of 0.852.

Original: In fact, the only additional metric that improves one’s ability to predict OOD accuracy is the average model confidence on ID data (more confident models have slightly better OOD accuracy), though this does not hold for LXMERT models (see Appendix Table 16).
Revised: In fact, the only additional metric that improves one’s ability to predict OOD accuracy is the average model confidence on ID data (more confident models have slightly better OOD accuracy), though this does not hold for LXMERT models (see Appendix Table 16).

Original: These results cast doubt on the value of RRR metrics.
Revised: These results cast doubt on the value of RRR metrics.

Original: If ID accuracy on its own is a better predictor of OOD accuracy than RRR metrics, then RRR metrics may not be a better measure of the quality of model reasoning than ID accuracy is.
Revised: If ID accuracy on its own is a better predictor of OOD accuracy than RRR metrics, then RRR metrics may not be a better measure of the quality of model reasoning than ID accuracy is.

Original: We believe the RRR metrics considered in this paper are still theoretically justified as model desiderata, but we cannot recommend them as measures of model generalization to OOD data.
Revised: We believe the RRR metrics considered in this paper are still theoretically justified as model desiderata, but we cannot recommend them as measures of model generalization to OOD data.

Original:  Discussion & Conclusion
Revised:  Discussion & Conclusion

Original: Limitations.
Revised: Limitations.

Original: Though we evaluate with two standard model architectures and three datasets, our conclusions may be limited to settings using Faster-RCNN [45] bounding box representations as the feature space rather than pixel space.
Revised: Though we evaluate with two standard model architectures and three datasets, our conclusions may be limited to settings using Faster-RCNN [45] bounding box representations as the feature space rather than pixel space.

Original: Additionally, though we follow existing guidelines with our distribution shifts [11, 59], we do not measure model generalization across all typical kinds of shifts [44].
Revised: Additionally, though we follow existing guidelines with our distribution shifts [11, 59], we do not measure model generalization across all typical kinds of shifts [44].

Original: Lastly, we note that FI supervision methods are limited by the need for additional annotations.
Revised: Lastly, we note that FI supervision methods are limited by the need for additional annotations.

Original: Ethics.
Revised: Ethics.

Original: We hope that our findings regarding model accuracy and explanation plausibility/faithfulness will help dispel the notion that models reason like humans (or are more grounded) simply because model explanations look similar to human explanations, which can cause unwarranted trust in ML models [24].
Revised: We hope that our findings regarding model accuracy and explanation plausibility/faithfulness will help dispel the notion that models reason like humans (or are more grounded) simply because model explanations look similar to human explanations, which can cause unwarranted trust in ML models [24].

Original: We do not foresee specific ethical risks arising from this work that do not already apply to the general use of machine learning for visual question answering tasks, such as the potential deployment of ML models in settings where they may harm people [62, 55].
Revised: We do not foresee specific ethical risks arising from this work that do not already apply to the general use of machine learning for visual question answering tasks, such as the potential deployment of ML models in settings where they may harm people [62, 55].

Original: Conclusions.
Revised: Conclusions.

Original: In this paper, we show that (1) FI supervision can improve VQA model accuracy via our V IS FIS method, (2) accuracy improvements appear to stem from improving explanation plausibility specifically for faithfully explained data, (3) FI supervision can improve RRR metric performance, and (4) RRR metrics do not actually correlate well with OOD accuracy.
Revised: In this paper, we show that (1) FI supervision can improve VQA model accuracy via our V IS FIS method, (2) accuracy improvements appear to stem from improving explanation plausibility specifically for faithfully explained data, (3) FI supervision can improve RRR metric performance, and (4) RRR metrics do not actually correlate well with OOD accuracy.

Original: Acknowledgements
Revised: Acknowledgements

Original: We thank Jaemin Cho for helpful discussion of this work, as well as Derek Tam, Xiang Zhou, and
Revised: We thank Jaemin Cho for helpful discussion of this work, as well as Derek Tam, Xiang Zhou, and

Original: Archiki Prasad for useful feedback.
Revised: Archiki Prasad for useful feedback.

Original: This work was supported by ARO Award W911NF2110220,
Revised: This work was supported by ARO Award W911NF2110220,

Original: DARPA Machine-Commonsense (MCS)
Revised: DARPA Machine-Commonsense (MCS)

Original: Grant N66001-19-2-4031, NSF-AI Engage Institute DRL211263, and a Google PhD Fellowship.
Revised: Grant N66001-19-2-4031, NSF-AI Engage Institute DRL211263, and a Google PhD Fellowship.

Original: The views contained in this article are those of the authors and not of the funding agency.
Revised: The views contained in this article are those of the authors and not of the funding agency.

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] All claims made in the abstract and introduction are properly supported by our paper’s results and contributions.
Revised: [Yes] All claims made in the abstract and introduction are properly supported by our paper’s results and contributions.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 8.
Revised: [Yes] See Section 8.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Also see Section 8.
Revised: [Yes] Also see Section 8.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 3.
Revised: [N/A] 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The code and instructions for reproducing the main results are in supplement.
Revised: [Yes] The code and instructions for reproducing the main results are in supplement.

Original: The datasets are freely available online (urls in Appendix E).
Revised: The datasets are freely available online (urls in Appendix E).

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See Section 6 and Appendix E and F. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] See Section 6 and Appendix E and F. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] All main experiments are conducted with 5 random seeds.
Revised: [Yes] All main experiments are conducted with 5 random seeds.

Original: Hypothesis testing are conducted with 10k bootstrapping.
Revised: Hypothesis testing are conducted with 10k bootstrapping.

Original: See Section 7.
Revised: See Section 7.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix F. 4.
Revised: [Yes] See Appendix F. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] All creators of existing assets are properly cited.
Revised: [Yes] All creators of existing assets are properly cited.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] All assets are used in accordance with their licenses (listed in Appendix E).
Revised: [Yes] All assets are used in accordance with their licenses (listed in Appendix E).

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether
Revised: [N/A] (d) Did you discuss whether

Original: and how consent was obtained from people whose data you’re using/curating?
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [Yes] See Section 8.
Revised: [Yes] See Section 8.

Original: If you used crowdsourcing or conductedresearch with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: If you used crowdsourcing or conducted

Original: A Feature Importance Explanation Methods

Original: We briefly review several FI explanation methods and explain how they are used in this paper.

Original: These methods can be classified as gradient-based (1-2), attention-based (3), and perturbation-based (4-7).

Original: Note that when computing derivatives of model outputs for explanation methods, we use the logit of the predicted class rather than the predicted probability for purposes of numerical stability.

Original: Vanilla Gradient (VGrad) [49].

Original: This method offers an explanation of model behavior in terms of the gradient of the model output with respect to the input, ∇ x f θ ( x ) ˆ y .

Original: When computing scores for a bounding box vector representation, we sum up the gradient for each element.

Original: Expected Gradients (ExpGrad) [15].

Original: This method estimates the integral in Integrated Gradients [54] by Monte Carlo sampling in order to speed up computation

Original: and it uses the data distribution to obtain baseline inputs.

Original: That is, the explanation

Original: [Equation]

Original: is estimated with a single sample of α and x ′ ∼ D using the training dataset D .

Original: We consider alternative baselines x ′ later.

Original: [Table]

Original: Attention weights (AttWeight) [25, 63, 52].

Original: This approach treats attention weights in a model as an explanation of model feature importance.

Original: For the Up-Down model [2], we use its sole set of top-down attention weights, but early experiments suggest this is not an effective method and we do not explore it further.

Original: Leave-one-out omission (LOO) [32].

Original: An LOO explanation assigns a score to feature j as the difference in the function output on the original input and an input with feature j replaced.

Original: Any Replace function may be used with LOO.

Original: Hence ˜

Original: e j = diff (cid:0) f

Original: ( x ) , Replace( x,⃗ 1 − j ) (cid:1) where diff measures the difference in function outputs, and ⃗ 1 − j is the ones vector with element j set to 0.

Original: Keep-one-in omission (KOI).

Original: The complement of leave-one-out, this method scores each feature by computing the effect of replacing all features except that a given feature.

Original: SHAP [37].

Original: We use the model-agnostic Kernel SHAP method, a generalization of LIME which assigns scores to features by fitting a linear model on perturbations of an input in order to predict the effect of each feature perturbation on the model output.

Original: Specifically, Kernel SHAP obtains an explanation by solving a weighted regression problem where model outputs are predicted based on the presence of features in the input:

Original: [Equation]

Original: (8) where s is a random binary mask over features, x s = Replace( x, s ) , the “null” input x ⃗ 0 =

Original: Replace( x,⃗ 0) and π is the Shapley kernel [37].

Original: Any Replace function may be used with SHAP.

Original: Average Effect (AvgEffect).

Original: This method follows SHAP exactly except for the use of a regression.

Original: To estimate a feature’s importance, we aim to compute the expected difference between model outputs with that feature observed vs. replaced:

Original: [Equation]

Original: where s 1 and s 0 are versions of a random binary vector s where some element has been set to 1 in s 1 and 0 in s 0 .

Original: In practice this expectation is estimated via Monte Carlo sampling.

Original: As long as elements of s are sampled independently, this method gives the same result as Kernel SHAP when the number of samples is large, but results will differ when the sample size is small.

Original: B Replace Functions

Original: We explain the tuning process for Replace functions in this section.

Original: As the Replace function is used in both obtaining model FI and data augmentation, we tune the Replace functions using the Align and Align+Suff-Human objectives with the LOO explanation method, and we select the function with the highest average Dev set performance.

Original: For the full sequential tuning process across all hyperparameters, see

Original: Appendix F.

Original: We consider five different Replace functions: All-Zeros, AllNegative-Ones, Gaussian, Marginal Distribution, and Shuffling.

Original: The first two functions simply replace features with zeros or negative ones.

Original: Gaussian function adds zero-mean Gaussian noise to input features with the standard deviation calculated using all features within the current batch.

Original: Marginal

Original: Distribution replaces a feature (a bounding box) with a randomly sampled feature (another bounding box) from the current batch.

Original: The Shuffle function shuffles elements of the input representation across all bounding boxes that need replacement within one sample (within and across bounding boxes).

Original: We find that All-Negative-Ones Replace function has the highest average accuracy on the Dev set, and we use it for all situations where replacement is needed (see Table 6).

Original: C Differentiable SHAP

Original: In this section, we show how to differentiate through SHAP explanations while respecting the theoretical properties that SHAP explanations provide.

Original: In Appendix D, we discuss how to limit the computational burden of computing perturbation-based explanations during model training.

Original: Kernel SHAP [37] values are obtained via a weighted linear regression as follows:

Original: To explain a model f : X → Y , one defines a data distribution D s over binary feature masks for randomly replacing features with some reference value (denoted in our paper by the Replace operation).

Original: In SHAP, these reference values are either (1) randomly drawn from the marginal data distribution over that feature, or (2) preset by the user to a fixed value for all features.

Original: We choose the second option based on

Original: Replace function tuning.

Original: The closed-form solution for SHAP values is then given by a weighted least-squares regression [37]:

Original: [Equation]

Original: where the row vector S i is drawn from D s , W is a diagonal weight matrix with elements W ii =

Original: π ( S i ) , and Y i is the difference in function outputs on x S i and the “null” input, f θ ( x S i ) − f θ ( x ⃗ 0 ) .

Original: This formulation can also satisfy the additivity constraint that the explanation weights sum to the difference f θ ( x ) − f θ ( x ⃗ 0 ) .

Original: This is done by adding a “data point” S i that is all ones, with its weight

Original: W ii manually set to a large value.

Original: The resulting explanation is differentiable w.r.t.

Original: θ by virtue of being differentiable w.r.t.

Original: Y .

Original: D Varying Compute Budgets in Feature Importance Methods

Original: In Table 10, we show the performance of each FI method for improving CLEVR-XAI dev ID accuracy with UpDn.

Original: Surprisingly, we find that accuracy improvements do not increase with a higher compute budget for the FI method.

Original: Below, we describe how the compute budget can vary for each method.

Original: Vanilla Grad and Attention have invariable compute budgets, as measured in terms of the number of forward+backward passes.

Original: However, the other methods have variable budgets.

Original: Expected Gradients depends straightforwardly on the number of sampled α values, since we use the same negative-ones baseline feature value for all points (one forward and one backward pass per sample).

Original: To compute a perturbation-based explanation, we need to compute f θ at least once per feature in x .

Original: This is because we need to measure the effect of replacing each feature separately.

Original: With SHAP, we need at least one sample S i per feature in order for ˜ e to be identifiable (equivalently, for S T WS to be invertible).

Original: However, a complete explanation is not needed for every datapoint during training .

Original: Instead, we can estimate feature importance for only a subset of features for each datapoint in a given batch.

Original: This allows us to greatly limit the computational cost of explanation supervision.

Original: In fact, we can use as little as a single sample per data point .

Original: The strength of SGD-based training in this context is that, over the course of training, a large number of feature importance estimates will be computed and penalized against human explanations.

Original: With our per-explanation compute budget of k model forward+backward passes and an input dimensionality d , we allow for k <

Original: d

Original: by explaining only k features while keeping the other d − k features constant.

Original: With LOO explanations, this simply requires not computing scores for d − k features, which are ignored in the L align loss.

Original: With SHAP explanations, we pick d − k features to always set to 0 in our random masks s .

Original: Then when computing Eq.

Original: 10, we drop those constant feature columns from S to obtain a new k × k matrix, which ensures that ˜ e is identifiable.

Original: E Data Details

Original: Dataset License.

Original: We conduct experiments on three datasets: CLEVR-XAI [5] under the CC

Original: BY-NC-ND 4.0 license, GQA [21], and VQA-HAT [11] both under the CC BY 4.0 license.

Original: Distribution Shift Resplit Sensitivity.

Original: Since we randomly construct ID and OOD splits with our distribution shift, we show the robustness of V IS FIS across three resplits of CLEVR-XAI dataset here.

Original: In Table 7, we see that V IS FIS gives significant performance improvements in all resplits.

Original: The absolute OOD accuracies vary across resplits, but the size of the OOD performance improvement

Original: [Table]

Original: between V IS FIS and the baseline is generally similar, with between a 2.3 and 3.

Original: Threshold for Human Feature Importance.

Original: For all our objectives except for Align, we need to select the threshold for human FI to separate important features from unimportant ones.

Original: We select the threshold separately for each dataset mainly based on (1) qualitative visualizations of the important features and (2) the percentage of data without important features.

Original: If a data point is without important features given a threshold, we do not use FI supervision objectives for that datapoint, but we do compute the main task objective, L Task .

Original: Although we want the importance features to be reasonable given qualitative visualizations, we don’t want to exclude too much data from training.

Original: We balance between good qualitative results and relatively few excluded data points by selecting thresholds of 0.85, 0.55, and 0.3 for CLEVR-XAI, VQA-HAT, and GQA respectively.

Original: These thresholds exclude 1%, 1%, and 8% of data from training with additional objectives for the three datasets.

Original: To ensure that

Original: V IS FIS is robust to this choice of threshold, we measure its performance improvement across a range of thresholds using UpDn on CLEVR-XAI.

Original: In Fig.

Original: 4, we present model accuracy as a function of the percentage of objects across images that are deemed as important based on a threshold.

Original: The values of the threshold vary from 0.1 to 0.98.

Original: The results show that performance improvements in ID and particularly OOD test accuracy are obtainable across a large range of threshold values.

Original: Training Size Ablation.

Original: GQA dataset [21] contains 943k training points and 132k validation points.

Original: After distribution shift, based on a ratio of 6:1:1.5:1.5, we obtain 645k, 107k, 161k, and 161k data for

Original: Train, Dev, Test ID, and Test OOD sets respectively.

Original: We then downsample the train set to about 1/ of its original size.

Original: We also exclude a small fraction of data with no ground-truth bounding boxes, and we limit our dev and test sets to 20k points.

Original: Thus the final split sizes are 101k train points, 20k

Original: Dev, 20k ID Test, and 20k OOD Test.

Original: We term this dataset GQA-101k in the main paper.

Original: To measure how FI supervision improvements vary with the amount of training data, we compare V IS FIS with the baseline for GQA using between 5k and 600k training points.

Original: Shown in Fig.

Original: 5, the results suggest that supervision is most helpful for improving OOD accuracy when using between 10k and 300k training points, though improvements in OOD accuracy may still be obtained beyond this value.

Original: [Table]

Original: Categorizing Faithfulness into Worst/Middle/Best Groups.

Original: As part of our analysis of how accuracy varies with explanation plausibility, we group datapoint explanations into three faithfulness categories, Worst, Middle, and Best.

Original: We select these based on theoretically sensible values of the

Original: Sufficiency and Comprehensiveness metric (see Sec.

Original: 5 for metric definitions).

Original: To be in the Best

Original: Sufficiency category, the average Sufficiency score (across explanation sparsity levels) must be at or below 0.01, meaning that the Replaced input must receive a predicted probability no more than one percentage point below the original.

Original: For UpDn on CLEVR-XAI, this is about 53% of the data.

Original: To be in the Best Comprehensiveness category, removing the top features must lower the predicted probability by at least 0.4 points (on average across explanation sparsity levels).

Original: We give the remaining values and data proportions in Table 8.

Original: F Training Details

Original: Our implementations makes use of PyTorch [42].

Original: Our UpDn model is optimized with a standard

Original: Adam [29], and LXMERT uses Adam with a linear-decayed learning-rate schedule [12].

Original: We use a batch size 64 for UpDn and 32 for LXMERT.

Original: For all experiments, we train UpDn for 50 epochs and

Original: LXMERT for 35 epochs.

Original: UpDn is trained from scratch, while LXMERT uses the default pretrained checkpoint.

Original: It takes about an hour to train UpDn on an Nvidia RTX 2080 Ti and about 6 hours for

Original: LXMERT on an Nvidia A100.

Original: Hyperparameter Tuning.

Original: We detail the tuning steps here.

Original: All tuning is done using CLEVR-XAI.

Original: The tuning is done in sequential order.

Original: We first tune learning rate for the baseline UpDn and LXMERT models.

Original: Learning rate is chosen from {1e-2, 5e-3, 1e-3, 5e-4, 1e-4} for UpDn and {5e-4, 1e-4, 5e-5, 1e-5}.

Original: We settle with 1e-3 and 5e-5 respectively.

Original: We then fix the learning rate and tune the weight λ

Original: i for different objectives.

Original: For augmentation objectives, we tune the weight with UpDn and use the same weight for LXMERT.

Original: The weight for augmentation is chosen from {100, 10, 1,

Original: [Table]

Original: [Table]

Original: 1e-1, 1e-2}, and we end up using weight of 1 for all augmentation objectives.

Original: For Inv-FI and Align objectives, we use FI method LOO with all-zeros replacement function, and tune the weight for

Original: UpDn and LXMERT separately.

Original: For UpDn, the weight is chosen from {100, 10, 1, 1e-1, 1e-2} and for LXMERT, it is chosen from {1e-3, 1e-4, 1e-5, 1e-6, 1e-7}.

Original: We use weight 1 for UpDn and weight 1e-3 for LXMERT+Inv-FI and weight 1e-6 for LXMERT+Align.

Original: We also tune the alignment function - Cosine Similarity, KL divergence, L1 distance, and L2 distance - for the Uncertainty and

Original: Align objectives and use KL for Uncertainty and Cosine Similarity for Align.

Original: In addition, we tune the weight for HINT [47] and SCR [64] with Vanilla Gradient.

Original: The weight is chosen {10, 1, 1e-1, 1e-2, 1e-3, 1e-4} for UpDn and {1e-3, 1e-4, 1e-5, 1e-6, 1e-7} for LXMERT.

Original: We use 1e-3 for UpDn and 1e-6 for LXMERT.

Original: We then fix the objective weights and tune the Replace function (results in Table 6).

Original: Finally, we tune the FI method to use for Inv-FI and Align objectives.

Original: KOI-gt works the best with Inv-FI, and Expected Gradient-pred for Align.

Original: The numbers for Inv-FI and Align in

Original: Table 11 are obtained with KOI-gt and Expected Gradient-pred respectively.

Original: We then tune V IS FIS with KOI-gt, Expected Gradient-pred, and, for fair comparison with other relevant works, Vanilla

Original: Gradient-gt.

Original: It turns out that Vanilla Gradient gives the greatest performance gain, and we choose it for all our experiments with V IS FIS (see Table 9).

Original: Stop Gradient.

Original: When backpropagating through model explanations, we apply a stop gradient for particular FI supervision methods in order to avoid influencing how the model handles the full input (which should be used principally for the task loss L Task ).

Original: For FI methods that involve baseline output f θ ( x ) or "null" output f θ ( x ⃗ 0 ) , which includes Excepted Gradient, LOO, KOI, and SHAP, we stop the gradient at f θ ( x ) and f θ ( x ⃗ 0 ) .

Original: G Additional Results G.1 Which Objectives Are Affected by Random Supervision?

Original: Design.

Original: In earlier experiments, we find that V IS FIS does not improve performance with random supervision.

Original: Here, we further explore how each of the four additional objective terms in V IS FIS is individually influenced by random supervision.

Original: To assess the effect of random supervision on each objective, we give random supervision to one of the objectives and normal supervision to the other three on CLEVR-XAI with UpDn.

Original: Results. We show the results in Table 12. The Suff-Human objective is the main reason why V IS FIS does not work with random supervision.
Revised: research with human subjects...

Original: Uncertainty and alignment objectives with random supervision hurt the performance, but not as much as the sufficiency objective.

Original: Note that Suff-Human with random supervision is different from Suff-Random, which has different features mask out across

Original: [Table]

Original: [Figure]

Original: ExplanationFaithfulness

Original: BestMiddleWorst

Original: [Figure] [Table] the training process for the same sample. Here, Suff-Human with random supervision has the same (random) features masked out for the entire training process.
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: The invariance objective with random supervision does not hurt the performance at all.

Original: G.2 Accuracy-Plausibility Relationship Across Test Splits, Datasets, and Models

Original: In the main paper Fig.

Original: 2, we show how accuracy varies as a function of explanation plausibility and faithfulness for UpDn models on CLEVR-XAI, and we group data points across ID and OOD test splits.

Original: Here, we show that the main trends are generally consistent across the choice of explanation metric (Sufficency vs. Comprehensiveness), test split (ID vs. OOD), dataset, and model.

Original: Trends across metric and split are shown in Fig.

Original: 6, and trends across datasets are shown in Fig.

Original: We show results for LXMERT on CLEVR-XAI in Fig.

Original: Though the trends weaken slightly in certain settings, we always find that accuracy correlates positively with plausibility for highly faithful explanations, while the relationship is weaker or non-existent for unfaithful explanations. [Figure] ExplanationFaithfulness BestMiddleWorst
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: Datapoint-level Accuracy vs. Plausibility

Original: [Figure] G.3 Which FI Method Produces the Most Faithful Explanations? Design. We calculate the Explanation Sufficiency and Explanation Comprehensiveness metrics for the FI methods listed in Appendix A, using either the predicted or ground truth class to select the output logit that is explained.
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: All experiments are conducted on CLEVR-XAI with UpDn.

Original: Following guidelines from Hase et al.

Original: [Figure]
Revised: [N/A]

Original: the UpDn models are trained with Suff-Random objective to make the replaced features in-distribution for the models.

Original: For LOO and KOI, we use a budget of 15 and 36 on CLEVR-XAI and VQA-HAT/GQA, which is the same number as the number of bounding

Original: [Figure]

Original: _ c

Original: [Figure]

Original: l o r e qu a l _ i n t

Original: e

Original: g e r e qu a l _ m a t

Original: e

Original: r

Original: i a l e qu a l _ s h a p e e qu a l _

Original: s i ze

Original: e x i s t g r ea t e r _ t

Original: h a n l e ss _ t

Original: h a n qu e r y_ c

Original: o l o r qu e r y_ m a t

Original: e r i a l qu e r y_ s h a p e qu e r y_

Original: s i ze

Original: Question Type

Original: OOD

Original: A cc u r ac y

Original: [Figure]

Original: BaselineVisFIS

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: boxes.

Original: For SHAP, Average Effect, and Expected Gradient, we use a budget of 1000 to reduce noise in deriving each explanation, as these methods involve random sampling.

Original: We select the best explanation method for each dataset by taking the best score on average across the two metrics.

Original: Results.

Original: We show the results in Table 13.

Original: In general, explanations obtained on predicted class are more faithful to the model decisions than those obtained on ground truth class.

Original: UpDn attention,

Original: LOO-pred, and KOI-pred are among the best across three datasets.

Original: SHAP and Average Effect surprisingly are not very faithful across all three datasets.

Original: KOI on predicted class is the most faithful one for CLEVR-XAI and VQA-HAT, while LOO on predicted class is the best for GQA.

Original: Hence, when calculating explanation metrics, we use KOI on predicted class for CLEVR-XAI and VQA-HAT and LOO on predicted class for GQA.

Original: G.4 Can Explanation Supervision Improve Model Explainability?

Original: Design.

Original: To assess the effect of FI supervision on model explainability, we record faithfulness metrics using all of our CLEVR-XAI models.

Original: We then plot explanation Sufficiency and Comprehensiveness for each model (averaged across five seeds) to visualize the distribution of faithfulness scores.

Original: Results.

Original: We show results for each model in Fig.

Original: 10 (scores also listed in Appendix Table 11).

Original: We find that average explanation Sufficiency and Comprehensiveness scores lie along a Pareto frontier, shown by the gray line, which represents a trade-off between better Sufficiency and Comprehensiveness (models better in one metric are worse in the other).

Original: Generally, explanation supervision does not

Original: [Table]

Original: improve model explainability relative to unsupervised models, with the exception of the Suff+Unc objective.

Original: In the bottom right of the plot, this model demonstrates a better combination of Sufficiency and Comprehensiveness than other supervised or unsupervised methods, including Saliency-Guided

Original: Training [22].

Original: The Suff+Unc model is especially explainable likely because the Sufficiency objective encourages the model to rely on a small number of important features, while the Uncertainty objective encourages the model to become less confident when those important features are removed.

Original: G.5 How Can Models with Low Plausibility Achieve High Accuracy?

Original: Shown in Table

Original: [Table]

Original: we find that models with lower average plausibility show different conditional relationships than models with higher average plausibility, which helps explain why low-averageplausibility models can achieve similar accuracies to high-average-plausibility models.

Original: Low-averageplausibility models have low plausibility points with low faithfulness scores, meaning these points are still often accurately predicted and hence do not bring down the average model accuracy.

Original: Meanwhile, middle and high-average-plausibility models often have low-plausibility points with highly faithful explanations, meaning these points are often inaccurately predicted, offsetting any gains to average model accuracy that are achieved for points with both highly plausible and faithful explanations.

Original: [Table]

Original: G.6 Do RRR Metrics Predict OOD Generalization? Additional Datasets and Models

Original: We measure the correlation between RRR metrics (calculated with ID data) and OOD accuracy across a large set of models.

Original: We report results additional in Table 16 here for LXMERT models on CLEVR-XAI and UpDn for GQA/VQA.

Original: We perform a cross-validation resampling model-level statistics 10k times, using 40 models’ metrics as training data and 5 for testing each time.

Original: The final metrics we consider are: (1) ID accuracy on its own as a baseline, (2) RRR metrics on their own, (3) ID accuracy plus average model confidence, (4) ID accuracy plus explanation metrics, (5) ID accuracy plus RRR metrics, and (6) All Metrics, which uses all available metrics.

Original: The results are similar to in the main paper, showing that RRR metrics do not achieve a better correlation with OOD accuracy than ID accuracy does on its own.

Original: [Table]