Paper: Towards Faithful Neural Network Intrinsic Interpretation with Shapley Additive Self-Attribution (ICLR-2024)

Reviewer comment (Theory):
I have my doubts as to how meaningful the learned marginal contribution scores from $\Delta$ can be in general. The goal is to have $\Delta$ for a single feature $i$ match its true Shapley score, but in the absence of true Shapley labels, the network is regularized to satisfy the following: 1) the summation of the $\Delta$ values over all features $i$ should be the label $y$ (Equation 3); 2) for any subset of features $\mathcal{S}$ which include $i$, different permutations of $\Delta$ on that subset should be the same value (Equation 4); and 3) partial sums of $\Delta$ values should also equal the label $y$ (Equation 5). As the authors point out, while the first two conditions are sufficient to ensure the model learns permutation-invariant contribution scores which add to the final label $y$, they alone are not sufficient to ensure that the contribution scores for each feature are meaningful (e.g. the feature contributions can be 0 for every feature except the last one). Condition 3 is meant to fix this, but it is not clear how that is done.

Author response:
The purpose of our distillation loss is not to make $\triangle$ match the Shapley value, but rather to make the learned attribution value converge to the average of our intermediate value $\triangle$ (Proposition 3.3). This will eventually make the attribution value the Shapley value of the final output (Theorem 3.4). All our losses are direct training losses instead of regularizers.

The statement in condition 3, "partial sums of values should also equal the label $y$," is misunderstanding. In terms of the physical meaning of $\triangle$ and $f_c$, the objective of Equation 5 is to make the sum converges to the expectation of $y$ given these observable features (i.e., the conditional probability $p(y|x_S)$ in binary classification), which is not the label of one sample but the average in the entire dataset. This is proven in Appendix Proposition D.1. In the scenario you mentioned, "feature $x_1$ does not matter" statistically means its observation does not influence the posterior distribution of $y$. Consequently, the conditional probability $p(y|x_1)=p(y|\emptyset)$, and the contribution of $x_1$ in this case is exactly 0 because $p(y|\{x_1\})=\triangle(x_1, \emptyset) + \phi_0$ and $p(y|\emptyset)=\phi_0$. 

In Appendix Figure 4, we also provided a plot of the model prediction using partial observation, which demonstrates that the model can make meaningful predictions with partial observations. Our paper rigorously proves that the Shapley module converges to the desired Shapley value, which demonstrates the effectiveness of these modules.

Reviewer comment (Theory):
In particular, condition 3 (Equation 5) seems like it trains the network to make sure that the partial sums of contribution scores (i.e. for any subset of feature $\mathcal{S}$) to be equal to the label $y$ _as if_ it had the full set of features. For example, consider a 2-feature dataset where feature 1 does not matter and the true label is simply equal to feature 2. Then in Equation 5, $L_v$ on just feature 1 would try and make the importance of feature 1 match the label, but in reality the contribution of feature 1 should just be 0.

Author response:
The statement in condition 3, "partial sums of values should also equal the label $y$," is misunderstanding. In terms of the physical meaning of $\triangle$ and $f_c$, the objective of Equation 5 is to make the sum converges to the expectation of $y$ given these observable features (i.e., the conditional probability $p(y|x_S)$ in binary classification), which is not the label of one sample but the average in the entire dataset. This is proven in Appendix Proposition D.1. In the scenario you mentioned, "feature $x_1$ does not matter" statistically means its observation does not influence the posterior distribution of $y$. Consequently, the conditional probability $p(y|x_1)=p(y|\emptyset)$, and the contribution of $x_1$ in this case is exactly 0 because $p(y|\{x_1\})=\triangle(x_1, \emptyset) + \phi_0$ and $p(y|\emptyset)=\phi_0$. 

In Appendix Figure 4, we also provided a plot of the model prediction using partial observation, which demonstrates that the model can make meaningful predictions with partial observations. Our paper rigorously proves that the Shapley module converges to the desired Shapley value, which demonstrates the effectiveness of these modules.

Reviewer comment (Evaluation):
The traditional challenges with computing Shapley values are: 1) how to tractably estimate the marginal contribution of feature $i$ when the number of subsets of features is exponential; and 2) given a subset of features, how to obtain a value/prediction when most models are not capable of handling partial inputs. SASANet addresses these challenges by effectively training over many subsets, thereby forcing the model to be robust to fewer features, across different subsets. This might lead to significant inefficiencies in training, given the possible subsets and orderings, especially as the number of features becomes larger. The time analysis in section 4.4 is appreciated, but that seems to only include the time taken to compute the feature importances _post hoc_. How does the training time of SASANet compare to other methods?

Author response:
Modeling predictions under partially observed results is indeed a more challenging task. Our permutation-based training approach requires 2-3 times more training epochs to converge compared to directly fitting the final results. However, we believe that the benefits of obtaining accurate Shapley values, being able to model non-linear feature-label correlation, and having high robustness in predicting with missing features outweigh the additional training cost. We will try to find a space to clarify this point in our paper.

Reviewer comment (Experiments):
In this work, SASANet is only being applied to a few small tabular datasets. Especially given the potential brittleness of the marginal contributions and the potential inefficiencies (as described above), it is possible that this method does not translate well to larger tabular datasets (e.g. gene expressions) or non-tabular datasets (e.g. images, text).

Author response:
In SASANet, our main objective is to model complex non-linear correlations between semantically meaningful features and labels, which is crucial for scientific analysis and discovery. The Shapley value provides a theoretically sound approach for evaluating this correlation. However, its intricate definition makes computation highly complex and involves permutations, making it challenging to handle large-scale features effectively. In the case of high-dimensional inputs like images, text, or gene expressions, where semantics depend on spatial interactions of multiple input points, the individual points (e.g., a single pixel) may lack specific conceptual meaning to explicitly contribute to the ouput. In this case, it is more reasonable to extract concepts from raw features using backbones and attribute concept contributions to the output instead of assigning an accurate Shapley value to each input point. To achieve explainable concept-based inference tasks for CV/NLP, we simply add an attribution layer on top of the concept extraction layer. Notably, depite broad application scenarios, our paper focuses on experimenting with tabular data. Tabular data naturally have explicit concepts with clear physical meanings, thus make the pure attribution ability more explicit to observe.

Reviewer comment (Writing):
Minor typos:  
- Shapley is spelled Shapely in several places  
- Section 3.2, Specially → Specifically  
- Typo in loss equation in 3.5

Author response:
Thank you for pointing them out! We will correct it.

Reviewer comment (Evaluation):
In the analysis of the accuracy of Shapley values, how were ground truth values obtained for RMSE?

Author response:
We draw different orders for each sample and directly calculate the Shapley value of the output based on its definition, i.e, sample orders and get the marginal contribution of each feature and get the average. In our experiments, we have added some engineering tricks to reduce the time complexity. For example, we pre-calculate and store the attention matrix and value among features. Then, we inrementally update the representations for each prefix subset in an order. However, it still took us days to get the ground truth results for these 1,000 samples.

Reviewer comment (Novelty):
The proposed network is not sufficiently interpretable, because the internal modules are still black boxes. This network makes predictions while automatically providing Shapley values of each input feature. However, the modules for computing marginal contributions and Shapley values are constructed as multi-head attention layers and learned by back-propagation. Therefore, the internal computation is still unexplainable. In comparison, the SHAPNet (Wang et al., 2021) makes all the layer-wise propagation of features in the network interpretable.

Author response:
SHAPNet calculates the Shapley value for each layer between hidden units, but it cannot accurately calculate the inputs' Shapley value to the final output. Our work's significance lies in being the first to realize global Shapley self-attribution, which is significant considering Shapley has been widely reconized as effective attribution metric. Indeed, attribution aims to accurately quantify the contribution of each input feature to the output, and the Shapley value is explicitly defined as the average change in output when adding a feature. It has a clear physical meaning in showing the effect of features without the need for a transparent intermediate procedure. Extensive post-hoc interpretations studies have been using the Shapley value itself as a model interpretation instead of discussing the mechanism of generating it.

Reviewer comment (Theory):
The loss function in equation (5) is confusing. It seems that the right-hand formula in equation (5) equals $L_m(x_S,y,O_S)$, because this term is independent of $(x’,y’)$. I check the proof in the appendix and I guess there is a typo: it should be $L_m(x_S,y’,O_S)$. In this way, $\sigma(f_c)$ is forced to model the distribution $p(y|x_S)$.

Author response:
Thank you for pointing this out and we will double check the notations.

Reviewer comment (Presentation):
The presentation of the paper needs improvement. The authors introduce the function of each module but do not provide an overview of the whole framework. Thus, when I read it the first time, it takes much time to understand the whole network. Besides, I suggest the authors clarify the models to be learned/optimized in each loss function. Otherwise, it may be confusing that whether equation (3) optimizes $\Delta(\cdot,\cdot;\theta_\Delta)$ or $f_c$.

Author response:
Thank you for your suggestion and we will polish our paper accordingly!

Reviewer comment (Theory):
What is the computational complexity in the inference stage? Does it only use the Shapley module for inference? If yes, then what is the benefit of learning an additional marginal contribution module? If no, then the complexity in the inference stage is still $2^n$.

Author response:
Only the Shapley module is used for inference. The purpose of learning marginal contribution is to converge the attribution value to the Shapley value of the final output, as shown in Theorem 3.4. Without learning marginal contribution and simply training a single Shapley module with a regularizer, we cannot obtain the Shapley value. This is because the optimization directions of the regularizer and prediction loss create a trade-off that makes it difficult to accurately converge to the constraint that the attribution value is the Shapley value of the final output.

Reviewer comment (Experiments):
Why are you learning a model to predict the marginal contribution $\Delta$ instead of directly predicting $f_c(x_S)$, which is used in KernelSHAP and Frye et al., (2019)? (Frye et al., 2019) Shapley explainability on the data manifold. In ICLR 2021.

Author response:
Both approaches are theoretically feasible. Modeling $f_c(x_S)$ directly and then taking the difference to obtain $\triangle$ is theoretically equivalent to our current method and does not affect convergence of $\phi$ to the Shapley value of $f$. However, in practical implementation, modeling $\triangle$ allows the model to focus more on  the unique contribution of each added input, which aligns with Shapley's definition. On the other hand, modeling $f_c(x_S)$ requires combining the information of the entire feature set and potentially make the effect of each feature vague. Therefore, we chose to directly model $\triangle$ to simplify the operation when we design our model.

Reviewer comment (Writing):
Typos: Section 3.5: $O_k,k$,$O_K$, $k=1$, and $O_i$ are all subscripts.

Author response:
Thank you for pointing this out and we will double check the notations.

Reviewer comment (Evaluation):
I don’t understand why comparing SASANet to interpretable approaches, more specifically to a linear regressor and a single decision tree. Those simple methods are sure to obtain somewhat deceiving performances when compared to a huge neural network, where having some black-box architecture is not a problem. Inherently interpretable predictors have advantages of their own (being fully transparent, for example) that can’t be simply quantitatively compared to others, and thus shouldn’t be compared solely in terms of performances. When it comes to interpretable machine learning, there is a huge literature on that matter, especially in an era where interpretability has been put forward for various reasons, so simple linear regressor and decision tree might just lead to underestimating the potential of interpretable approaches and overestimate the power of SASANet.

Author response:
> About interpretable approaches

Table 1 aims to highlight the advantages of SASANet over other self-attributing models. The two black-box models and classic interpretable models widely used in tabular data are reported to position the expressiveness of these self-attributing models. We are not trying to prove SASANet is superior to self-interpreting models under any other interpretability paradigms. Since different paradigms have different goals and formulations, we believe they cannot be compared.

Under attribution paradigm, our focus is on accurately quantifying the contribution of features to the output, which has its unique significance for feature selection, debias, and discovering feature-outcome correlations. We believe that our current paper effectively demonstrates how SASANet excels other self-attribution models with its  well-grounded theoretical basis and high accuracy.

Reviewer comment (Experiments):
I don’t feel like the comparison to black-box models is fair. Was the MLP of reasonable size? No detail is shared on that matter. Comparing an MLP to a richer and more complex architecture involving transformers is questionable. Also, LightGBM dates a bit (2017); the goal of those experiments is to compare SASANet to state-of-the-art approaches when it comes solely to performances (i.e. black-boxes) but I don’t feel like those baselines are sufficient.

Author response:
> About black-box models

We use these classic black-boxes commonly used for tabular data as references to demonstrate our superiority over other self-attribution methods. We have never claimed SASANet to achieve state-of-the-art black-box models' performance. It is clear that other self-attribution methods have struggled to match MLP and LightGBM, while SASANet does, showing it significantly enhances usability with guaranteed interpretability but acceptable sacrifice on expressiveness, at least nearly reaching MLP. We did not compare transformers because they are not common for tabular data. In terms of MLP size, as stated in the paper, we fine-tuned them to their best performance, since simply training an MLP has weaker supervision than SASANet's loss and more easily overfit. For instance, we find the best performance is obtained by $4 \times 64$ for CENSINCOME and $4 \times 512$ for HIGGS, which can be found in our code and we will summarize them in appendix later.

Reviewer comment (Evaluation):
Fidelity experiments: It is stated in the abstract that « SASANet is shown more precise and efficient than post-hoc methods in interpreting its own predictions ». I don’t feel like this has been demonstrated in any way. The protocol for this demonstration is explained as follows: « We observed prediction performance after masking the top 1-5 features attributed by SASANet for each test sample and compared it to the outcome when using KernelSHAP, a popular post-hoc method, and FastSHAP, a recent parametric post-hoc method. The results are shown in Table 7. » And the conclusion that is drawn is the following: « SASANet’s feature masking leads to the most significant drop in performance, indicating its self-attribution is more faithful than post-hoc methods. » There is a logical gap here. Much information remains unknown: Were the features considered by the approaches correlated (to those in the top-5) in any way? Were the importance given by each approach to their top-5 approximately the same? Why not remove, for example, the features accounting for the first 20% importance? Also, Shapley values measure the difference in prediction, not the difference in performance. Therefore, a feature could have a great impact on the predictions while not affecting the performances at all (or smaller than expected). E.g. having an error of -x instead of x, thus a same squared error. Considering The protocol is simply insufficient to claim that « SASANet is shown more precise and efficient than post-hoc methods in interpreting its own predictions ».

Author response:
> About Fidelity experiments:

In the feature-masking experiment, the correlation between a feature and the outcome is determined by the model itself, not the specific interpreter. Each interpreter's attribution is compared in terms of how good they distinguish more important features, whose removal results in larger performance reduction, remain consistent regardless of the interpreter used. Therefore, removing the same number of features and compare which interpreter makes a better selection is a fair and intuitive comparison. 
It is statistically unlikely that removing important features on which a trained model relies for accurate predictions make all samples deviate towards a prediction that maintains the original performance. In reality, once important features are removed, the model lacks sufficient evidence, and the prediction becomes more akin to guessing. A guess can be difficult to perform as good as a reasonable inference based on strong evidences. 

In addition to feature-masking experiment, Table 3 shows our fidelity in a different perspective, which directly indicates that we learned more accurate Shapley values than post-hoc methods.

Reviewer comment (Writing):
Typo in Theorem 3.4: « with ample permutation ».

Author response:
> Typos

Thank you for pointing out the typos! We will correct them in our paper.

Reviewer comment (Experiments):
When it comes to feature attribution approaches, the time is reported in Table 3, but how does the training time of SASANet compare to others?

Author response:
> Training time of SASANet 

Post-hoc methods usually require no training, while incur high-cost computation for every sample. FastSHAP has a slow training process, preventing us from obtaining results on the HIGGS dataset in Table 2.

Compared to regular training procedure that directly fits the final results, SASANet requires 2-3 times more epochs to converge since it learns additional information, such as Shapley values and predictions under partially observable features. However, we believe this trade-off is reasonable in practice, as it allows us to gain more information for both higher interpretability and insights into non-linear feature-label correlation, as well as an ability to robustly handle any severe missing feature problem.