Paper: Exploring Unified Perspective For Fast Shapley Value Estimation (ICLR-2024)

Reviewer comment (Novelty):
The main contribution of this paper is SimSHAP, which involves training a Shapley value estimation model by comparing to estimated values for each gradient step. While there is a lot of build-up to this proposal, this idea has been explored by at least two other works [1, 2], so the novelty is limited. These works are not cited, and I'm not sure there's much methodological innovation on top of them.

Author response:
Thanks for reviewer's suggestions, we have added two references to the revised version.
- **About missing references:** We also noticed these works in our methodological survey, and we acknowledge that there have been some research efforts on amortized estimators in recent years, such as the FastSHAP method cited in our paper. 
- **About contributions:** We apologize for the unclear parts in the paper, but we still want to emphasize the main contributions of this paper. 
Many recent works have studied different estimation methods for approximating Shapley values, such as **semi-value** (ordinary Monte Carlo estimation) and **least-square value** (using the weighted least squares property of Shapley values). 
However, we found that these methods are **essentially not different**.
We find that the least-square value is essentially still equivalent to some special probability Monte Carlo estimator. 
Therefore, we propose a unified perspective of stochastic estimators and build a unified perspective of amortized estimators on this basis.
We have emphasized the main contributions of the paper in the revised version.

Reviewer comment (Theory):
Regarding the unified perspective in Section 2.3, it seems like this view is clearly apparent for the semivalue-based estimator and does not add much here. After reading the paper closely, I'm not sure it's correct for the least squares estimator. Previous work by Covert & Lee (2021) showed a similar derivation from the KKT conditions, and the analytic solution involves two terms that are estimated: one that resembles the importance sampling term, and one that resembles the transform term (see their section 3.2). In this work, it looks like the transform term $T$ is assumed to be known, which does not correspond to KernelSHAP but rather an alternative version from that work called "unbiased KernelSHAP." Are the authors perhaps analyzing that version rather than the original one? Note that the original one cannot be calculated without a sufficient number of samples due to invertibility issues (the authors here manage to use a very small number in their experiments, suggesting the unbiased version), and that the original one is not provably unbiased (whereas the authors claim zero bias here in eq. 14).

Author response:
Sorry for the confusion in the original paper. 
- **Least-square value & unified perspective:** Regarding whether the least-square value can be incorporated into our unified perspective, we have provided detailed analysis and rigorous proof in **Appendix A.1**.
- **About unbiased kernelSHAP:** Thanks to the reviewer for pointing out that we did use the **unbiased kernelSHAP** in this work, and we have made it clearer in the revised version to eliminate any possible confusion.
- **Proof of Eq(14):** Regarding the proof of the unbiasedness of the original formula 14, we have provided detailed derivations in **Appendix A.2** and ensured its unbiasedness.

Reviewer comment (Theory):
Regarding the unified view of amortized estimators, it is true that you can compare to the true Shapley values with any properly defined metric and learn the correct function. It is therefore not surprising that you can use the l2 distance. The authors might have been more careful about showing why it's okay to train on inexact values from the least squares estimator, because these training targets will be very noisy. What is the impact of this noise during training? Also, note again that [1, 2] have already explored this approach.

Author response:
Thanks for the suggestion. 
- **About the fitting priority of information with different frequency:** We acknowledge the need to clarify how the model can be trained on noisy data. We have added the analysis in **Appendix A.4**. 
- **Supplementary exprimental results:** The model can ensure a fit to the true value if the training set is large enough. 
As the reviewer pointed out, a small dataset can cause the model to overfit the noise and fail to capture the true value. We plan to research ways to learn the true value with limited data as our future direction. We have demonstrated the above statement with experiments in **Appendix A.7**. Furthermore, our sampling strategy guarantees convergence to the ground truth value when operating on sufficient amount of data. Together with Table 12 in **Appendix A.6**, we can ensure the feasibility of our algorithm.

Reviewer comment (Theory):
Regarding the unified view of amortized estimators, I'm not sure the proposed view of FastSHAP is correct. For the unnumbered equation defining $\mathcal{L}(\theta)$ in Section 2.3, there are two equations and I'm not able to see whether they are mathematically equivalent. The authors should consider adding this derivation, either here or in the appendix. Furthermore, the expression for $\phi_x$ is not equal to the Shapley values, because it isn't the solution from the KKT conditions and doesn't incorporate the constraint. Currently, I do not see how this perspective is correct, and it doesn't seem helpful to view the FastSHAP loss as comparing to the true values under a complicated metric: it misses the main point of the FastSHAP loss, which is that it bypasses the need for the ground truth. This perspective is also not necessary to understand that comparing to the ground truth using l2 distance should work.

Author response:
- **About the theoretical derivation of Eq(13):** We are sorry to have deleted the theoretical derivation due to the page limit. We can ensure its rightness and have included the whole proof in **Appendix A.3**. 
- **About $\phi_x$'s analysis:** In addition, we admit the inaccuracy of $\phi_x$ to Shapley values. That's why we add the $\frac{v(N) - v(\phi)}{d}\textbf{1}$ term to make it strictly equal to Shapley values, which is also a technical difference between **SimSHAP** and **FastSHAP**. 
  - **For FastSHAP:** FastSHAP utilizes normalization/penalty punishment terms in the loss to ensure correctness. 
  - **For SimSHAP:** We have found that the issue can be resolved by simply incorporating this term into the calculation of the sampled ground truth, allowing for the construction of SimSHAP without the need for normalization or penalty in the end.
- **About the main point of FastSHAP:** We acknowledge its contribution by bypassing the need for the ground truth labels. Here we would like to reiterate our contribution in unifying and proposing SimSHAP: 
  - for stochastic estimator (Table 1), we demonstrated that the least squares value with KKT condition is equivalent to direct Monte-Carlo sampling on the classic form of Shapley values computation. This serves as the answer to why we didn't discover a significant difference in computational complexity (see **Appendix A.1**). 
  - What's more, We would emphasize that SimSHAP bypasses the need for ground truth Shapley values labels as well, but we choose the simplest metric and function as a result of our unified framework.

Reviewer comment (Presentation):
Related to the above point, I'm not sure the entries for FastSHAP in Table 2 are correct.

Author response:
We have addressed the questions above, primarily on W4.

Reviewer comment (Evaluation):
In Figure 2, SimSHAP does not appear to outperform FastSHAP. In fact it's noticeably worse for the census dataset. Why is that, and can it be made more accurate?

Author response:
Thanks for the insightful suggestion. 
- The results in **Appendix A.6-7** demonstrate that significantly enlarging the sample size can notably enhance the model's fitting performance, but it also introduces additional computational burdens.
- To maintain consistent training times compared to the baseline approach, the results depicted in **Figure 2** do not incorporate a large sample size. Future research endeavors will focus on exploring how to achieve superior fitting outcomes with smaller sample sizes, as outlined in **W3 in part 1**.

Reviewer comment (Evaluation):
The results in Table 3 appear to contradict those in Figure 4. How can SimSHAP have the best AUC when its curves are far from the best?

Author response:
We're double-checking the details of this experiment and sorry for the late reply. Previously, we normalize the score curve to begin with 0 for insertion (1 for deletion) and end with 1 for insertion (0 for deletion).  While computing the mean and standard deviation, we only satisfy that the network output for the **original image (grand coalition)** is the same. That's why inconsistency is posed. We only enforced the latter constraint now and we can ensure the consistency between Figure 4 and Table 3.

Reviewer comment (Novelty):
The description of related work is incorrect in several places. In the introduction, "semivalue" and "least squares value" are not estimation methods, they are classes of game-theoretic solution concepts, and their mathematical perspectives are used to develop estimation methods (e.g., fitting a weighted least squares model with an approximate version of the least squares objective). Also in the introduction, model-specific methods do not "reduce computation cost by merging or pruning redundant feature combinations," this is not true for TreeSHAP, LinearSHAP or DeepSHAP. It is also not true that "the exact differences among these algorithms remain unclear," the differences seem very clear, as they are described in this work. The gap in missing work seems to be that the fastest solutions either are not general enough (TreeSHAP) or accurate enough (FastSHAP), although this work does not appear to solve the latter problem.

Author response:
We admit our incorrectness in the "Related Work" section and "Introduction" section. 
- Regarding terms such as semivalue, we follow the terminology used in [1].
- We have revised the paper and modified "reduce computation cost by merging or pruning redundant feature combinations" to "introduce model-specific information" in order to be more appropriate and accurate. 
- Moreover, we have to clarify "the exact differences among these algorithms remain unclear". Our motivation is that there are so many model-agnostic methods for Shapley values estimation and we sometimes don't know which method to choose for a specific scenario, so we turned to explore their differences in essence. We thus articulated this motivation in the new version of our paper.

Reviewer comment (Theory):
The "large numeral law" is not common terminology, it may be better to call this the "law of large numbers" as it is typically called. Also, the LLN does not imply an approximation error of $\mathcal{O}(1/\sqrt{M})$, it only implies convergence of the sample mean to the expectation. For the error rate, it might be better to cite a probabilistic error bound like Chebyshev's inequality.

Author response:
We apologize for not double-checking the official term for "law of large numbers". In addition, We are referring to the convergence of the sample mean to the expectation by mentioning $O(\frac{1}{\sqrt{M}})$, and we think the reviewer's suggestion makes the statement more precise.

Reviewer comment (Theory):
For proposition 1, it would be helpful to refer to a proof. Or even better, because this is a known result, the authors might refer to one of the works that originally showed it to be true, for example Charnes et al 1988.

Author response:
We have proofed **proposition 1** in **Appendix A.1**, and we also accept the advice to emphasize the result by citing the work the reviewer mentioned.

Reviewer comment (Writing):
In eq. 8 regarding FastSHAP, the authors' notation neglects to show the input $x$ in the cooperative game $v(S)$. It may be difficult for readers to follow what $v(S)$ represents. It would also help to give a specific example of how $v(S)$ is defined in Section 2.1, because no equation is currently provided.

Author response:
We apologize for any confusion in the notation system. 
We have organized the symbols in the new version.

Reviewer comment (Writing):
Regarding the additive efficient normalization, there are parts where FastSHAP is described as "biased" or that its predictions need to be rectified. The current description could be confusing to many readers, because it suggests that the loss is somehow incorrect, but that's not quite true. For example, you could write that the weighted least squares loss does not encourage the correct optimal value unless the predictions satisfy the efficiency constraint, which can be enforced by applying the normalization to the predictions.

Author response:
The reviewer is right as we did not intend to demonstrate the FastSHAP loss as a biased loss, but it needs to be normalized which corresponds to satisfying the efficiency constraint. It's also worth mentioning that we added one term instead of performing normalization for Shapley values correction in our algorithm, as stated in **W4** for weakness.

Reviewer comment (Writing):
There are small typos throughout the paper, for example "remarkbly" on page 3 and "minimizaion" on page 4. The paper could be more thoroughly proofread.

Author response:
We acknowledge that we followed FastSHAP's experimental setup to provide a systematic evaluation of our method. 
We have clearly stated this in the revised version of the paper and have also corrected all the typos pointed out by the reviewer, which has contributed to a more organized and precise presentation.

Reviewer comment (Novelty):
Overall, the paper's structure is extremely similar to the FastSHAP paper, to the point that the authors have replicated all the main text experiments, many of the supplementary experiments, and have nearly copied some of the writing. It would be helpful to either acknowledge this similarity, or deviate from it more significantly.

Author response:
We acknowledge that we followed FastSHAP's experimental setup to provide a systematic evaluation of our method. 
We have clearly stated this in the revised version of the paper and have also corrected all the typos pointed out by the reviewer, which has contributed to a more organized and precise presentation.

Reviewer comment (Writing):
Some typos and unclear passages makes the paper harder to read at times (see below).

Author response:
We apologize for any confusion caused. We have carefully proofread and modified the text based on the reviewer's feedback.

Reviewer comment (Writing):
Why did the authors define \( f:\mathcal X \to \mathcal Y \) in Sect 2.1 if this is not used in the rest of the paper? Note that the fact that \( f(x_S) = v(S) \) was never made.

Author response:
We recognize that there is room for improvement in our notations. We have streamlined the notation and removed any potential confusion.

Reviewer comment (Novelty):
The authors do a good job at commenting on related works for approximation of Shapley values, due to their complexity. However, there is no mentioning to works that provide tractable, exact or finite-sample approximate, computations of these values in cases where distributional assumptions are made:  

i) [Chen, Jianbo, et al. "L-shapley and c-shapley: Efficient model interpretation for structured data." arXiv preprint arXiv:1808.02610 (2018).]  

ii) [Teneggi, Jacopo et al. "Fast hierarchical games for image explanations." IEEE Transactions on Pattern Analysis and Machine Intelligence 45.4 (2022): 4494-4503]  

The authors should comment on these because, in these cases, approximations are not needed, and thus Shapley coefficients admit provably more efficient computations. Naturally, if these properties of the data are not met, the approximation strategies that the authors describe do provide useful estimation tools.

Author response:
Thank you for recommending additional related literature. We have incorporated more references on model-specific Shapley value estimation into our revised manuscript.

Reviewer comment (Theory):
In sec 2.3, the authors first define their Unified Stochastic Estimator and then show that it is a generalization of other schemes. However, their Definition 2 reads "Most existing stochastic estimators can be unified.." which sounds like a claim to be proven, not a definition. Indeed, this is shown/proven immediately after in the form of running text. I think the authors should consider re-organizing this Definition + Remarks as Definition + Theorem, which states the strict generalization.

Author response:
We agree that the previous presentation of **Definition 2** could be confusing. We have restructured this definition as: 
> **Unified Stochastic Estimator** is defined as the linear transformation of the values obtained from sampled subsets $S$.

Reviewer comment (Theory):
I think that to say that "Deep learning techniques have revolutionized various industries due to their learning capability by universal approximation theorem" is a stretch. Many methods provide universal approximation. The reasons for deep learning becoming so popular are others and more diverse.

Author response:
The reviewer raises an important point in this statement. We have revised the Introduction section to more clearly highlight the profound impact of deep learning techniques on various industries, due to their remarkable ability to learn complex functions both quickly and accurately.

Reviewer comment (Theory):
"the optimization object L in Eq. (5) need[s] to be approximated"

Author response:
We appreciate the reviewer's careful reading. The suggestions for details are helpful and we have modified our paper based on them.

Reviewer comment (Theory):
There is an \( \arg\min_\eta \) missing in the right-most term in Eq(5).

Author response:
We appreciate the reviewer's careful reading. The suggestions for details are helpful and we have modified our paper based on them.

Reviewer comment (Writing):
capaicity → capacity

Author response:
We appreciate the reviewer's careful reading. The suggestions for details are helpful and we have modified our paper based on them.

Reviewer comment (Writing):
In equation 14, remove " \(\to\) Shapley values" as this reads as "tends to", which is not what the authors meant presumably.

Author response:
We appreciate the reviewer's careful reading. The suggestions for details are helpful and we have modified our paper based on them.

Reviewer comment (Theory):
The proposed SimSHAP seems insufficiently motivated. Why did the authors make the specific choices for the stochastic estimator and metric space that they did?

Author response:
**Weakness 1 & Question 1:** 

We admit that the proposed SimSHAP is indeed theoretically
motivated. Our motivation is that based on the unified framework, we can choose the semi-value equation and identity matrix for their simplicity. In addition, as part of our future work, we aim to identify the optimal metric matrix that maximizes performance in terms of optimization convergence.

Reviewer comment (Evaluation):
It is not clear from the experiments if there are any performance benefits from SimSHAP beyond faster inference.

Author response:
**Weakness 2 & Question 2:** 

- **About highlighting our contribution:** In this context, we wish to **highlight our primary contribution of unification**: Numerous recent studies have investigated various estimation techniques for approximating Shapley values, such as applying semi-value (conventional Monte Carlo estimation) and least squares value (utilizing the weighted least squares property of Shapley values) equations. We discovered that current Shapley values estimation approaches are **essentially not different** and we found that the least squares value estimation is fundamentally still equivalent to a specific type of probability Monte Carlo estimator. As a result, we put forward a unified viewpoint of stochastic estimators and built a unified perspective of amortized estimators based on this concept.
- **About the choice between SimSHAP and FastSHAP:** With the unified framework, we proposed SimSHAP as a new perspective for Shapley value estimation, indicating its **comparable** performance to FastSHAP.
- **About the deletion curve:** With respect to the deletion curve in our updated version of the paper, the advantage of SimSHAP is not particularly pronounced. Nevertheless, we will continue to conduct thorough analysis of the outcomes as part of our future work.

Reviewer comment (Writing):
It would be helpful if the authors were a bit more explicit about notation, e.g. explicitly defining $\mathbf{J}$ in the definition of the least squares value transformation $\mathbf{T}$ and $\mathbf{1}^{\mathbf{S}}$ in the definition of the least squares value.

Author response:
**Other notes 1:** 

We are grateful to the reviewer for their careful reading and thoughtful suggestions regarding our notation. In our revised manuscript, we have included the definition of $J$ for clarity. Additionally, we have double-checked the definitions of $T$ and $1^S$ at the point where they first appear in the text to ensure consistency and clarity.

Reviewer comment (Presentation):
The insertion curve seems to have two green curves but no blue curve. There might be a plotting bug somewhere?

Author response:
**Other notes 2:** 

We have meticulously reviewed our code and experimental design. These revisions have been incorporated into the updated version of our paper.

Reviewer comment (Experiments):
Why did the authors design SimSHAP the way they did?

Author response:
**Weakness 1 & Question 1:** 

We admit that the proposed SimSHAP is indeed theoretically
motivated. Our motivation is that based on the unified framework, we can choose the semi-value equation and identity matrix for their simplicity. In addition, as part of our future work, we aim to identify the optimal metric matrix that maximizes performance in terms of optimization convergence.

Reviewer comment (Evaluation):
What are the reasons for using SimSHAP over FastSHAP? Is the primary benefit of SimSHAP faster inference time, or are there other benefits? Also, can the authors explain why SimSHAP outperforms FastSHAP specifically in the deletion AUC for CIFAR-10 and if there is a generalizable takeaway there?

Author response:
**Weakness 2 & Question 2:** 

- **About highlighting our contribution:** In this context, we wish to **highlight our primary contribution of unification**: Numerous recent studies have investigated various estimation techniques for approximating Shapley values, such as applying semi-value (conventional Monte Carlo estimation) and least squares value (utilizing the weighted least squares property of Shapley values) equations. We discovered that current Shapley values estimation approaches are **essentially not different** and we found that the least squares value estimation is fundamentally still equivalent to a specific type of probability Monte Carlo estimator. As a result, we put forward a unified viewpoint of stochastic estimators and built a unified perspective of amortized estimators based on this concept.
- **About the choice between SimSHAP and FastSHAP:** With the unified framework, we proposed SimSHAP as a new perspective for Shapley value estimation, indicating its **comparable** performance to FastSHAP.
- **About the deletion curve:** With respect to the deletion curve in our updated version of the paper, the advantage of SimSHAP is not particularly pronounced. Nevertheless, we will continue to conduct thorough analysis of the outcomes as part of our future work.

Reviewer comment (Writing):
What do the authors mean by "That’s mostly because of the requirement of number of mask is larger for SimSHAP" when explaining why SimSHAP training is slower than FastSHAP training in Section 4.2.4 (speed evaluation)?

Author response:
**Question 3:** 

We aim to demonstrate that we have recognized that SimSHAP requires a relatively large number of samples during the training process. As part of our future research, we intend to further investigate this phenomenon both theoretically and experimentally, which we believe may be related to the choice of the metric matrix.