Paper: Sum-of-Parts Models: Faithful Attributions for Groups of Features (ICLR-2024)

Reviewer comment (Novelty):
The work’s level of novelty can be made much clearer if the authors could spell out the similarity and difference between SOP and some existing methods.

Author response:
We have added a table to show the similarity and differences between SOP and other methods in Appendix A Table 2.

Reviewer comment (Reproducibility):
While the authors explain the high-level rationales of the method and evaluation, descriptions of some important details are either missing or too terse.

Author response:
Thank you for the suggestion of writing. We include more details for the method in Appendix C in the updated manuscript.

Reviewer comment (Evaluation):
For the case study, it is unclear if the insights were more from SOP or simply from combining a priori knowledge and the outcomes of the model to be explained.

Author response:
It is true that we can combine the a priori knowledge with the outcomes of any explainable model. However, the interpretation of different explanations are different. Our model gives faithful groups and their contributions to the final prediction, and the case study shows that the faithful groups correspond to the a priori knowledge, meaning that the semantically meaningful groups are contributing to the prediction.

Reviewer comment (Novelty):
Many methods share similar features with SOP’s Group Generator and Group Selector, so the authors should highlight similarities and differences to help readers judge novelty. In particular, more detail is needed comparing SOP to RISE and LIME, since RISE’s masks seem equivalent to SOP’s groups and LIME also learns weights on masked areas. Both methods appear to have elements resembling the Group Generator and Group Selector, and spelling out the similarity/difference would be helpful.

Author response:
Although RISE is also combining the masks, their prediction is not based on the outputs from the masks. So the post-hoc masks are not directly contributing to the prediction, where SOP's masks are directly used in making the prediction.

Reviewer comment (Writing):
The powerset P is not used in the equations. The reviewer questions if the last line of page 2 should read \Sum{S \in P} instead of just S.

Author response:
Yes. We apologize for the confusion. It should be $\sum{S \in P}$.

Reviewer comment (Theory):
In Definition 1, it is unclear why the difference is between \Delta f and \sum a, as these seem like very different quantities to compare. The same question applies to Definition 2.

Author response:
$\Delta f$ is the difference between predicting with the whole input x and x without subset $S$. \sum a is the sum of attribution scores for features in S. We would hope that attribution scores for each feature faithfully represent how much that feature contributes to the prediction of $f(x)$. Thus $f(x) - f(x_{\lnot S})$ should be similar to \sum a.

Reviewer comment (Presentation):
Clarification is needed on how the y-axis in Figure 2 is calculated. The scale is unexpected, as it was assumed f(x) gives a probability between 0 and 1.

Author response:
The total deletion error is $\sum_{S\in\mathcal{P}} \mathrm{DelErr}(\alpha,S)$ where $\mathcal P$ is the powerset of $\{1,\dots, d\}$, as in Definition 1. Thus y-axis is the sum over the whole powerset.

Reviewer comment (Theory):
The reviewer asks if the exponential increase in Figure 2 is simply a consequence of summing over a powerset of differences, where the size of the powerset grows exponentially with d.

Author response:
Yes.

Reviewer comment (Theory):
The paper claims a linear model achieves 0 error as defined. The reviewer asks whether this metric is only 0 for linear models, whether a non-linear model can have 0 error, whether SOP has 0 error, and suggests going through a simple calculation to illustrate.

Author response:
By definition here, only a linear model can achieve 0 error. SOP have 0 error. For SOP, we have access to the internal process of the new model, which is the linear combination of different passes of the old model. As a simple example, suppose $p(x) = x1 * x2 + x2 * x3$ where $x1, x2, x3 \in \{0, 1\}$, and SOP selects group $S1 = {x1, x2}$ and $S2 = {x2, x3}$, and assigns score $\alpha_1 = 1$, $\alpha_2 = 1$. Then, when $x1, x2, x3=1$, $InsErr = p(x_S) - p(0_d) - \sum_i \alpha_i = 2 - 0 - (1 + 1) = 0$. And if we only select $S1$, then $x1 = 1, x2 = 1, x3 = 0$, $InsErr = 1 - 0 - 1 = 0$. This is true for all cases. For images, this is also the same.

Reviewer comment (Theory):
Clarification is requested on in what sense SOP is more faithful than LIME, RISE, and GradCAM, as theorems 1 and 2 might address this but the reviewer remains unclear.

Author response:
SOP first generates the groups, and aggregates the separately obtained results from the groups. In this way, the prediction from each group is inherently based only on the group and different from post-hoc methods such as LIME, RISE, and GradCAM. Also the groups in SOP overcome the inherent limitation of individual feature attributions.

Reviewer comment (Reproducibility):
The reviewer asks how the parameters are learned, including the objective function and the learning algorithm.

Author response:
The extra attention layers in GroupGen and GroupSelect are learned end-to-end and the model is trained on cross-entropy for classification.

Reviewer comment (Writing):
More clarity is needed on the definition, construction, and intuition of the W’s, C’s, and the sparsemax.

Author response:
The W’s are learned end-to-end as part of the attention mechanism. The C is taken from the trained model’s classification layer, as it is the representation of the classes. The intuition behind this is to compute which group is most similar to each class and should be used for predicting that class. Sparsemax is used to completely zero-out some segments and some groups for a more human-interpretable set of groups.

Reviewer comment (Writing):
The meaning of the “, C z^T” (especially the comma) in Equation (4) needs clarification.

Author response:
$C z^T$ means doing matrix multiplication of the class weight matrix and the pooler output of each group, which is equivalent to $y_1, … y_G$. The comma means that there are two things outputted from GroupSelect. The first is the weights for each group for each class $c_1, …, c_G$. The second is the prediction from each group $y_1, …, y_G$.

Reviewer comment (Evaluation):
The reviewer suggests showing the curves for the insertion/deletion metric and the grouped versions.

Author response:
We have added insertion and deletion examples for grouped and standard cases in Appendix D.3.

Reviewer comment (Evaluation):
In the case study, it seems the Void and Cluster types are known beforehand, so it is unclear how SOP groups provided new insight. The reviewer also asks what advantage the Group Selector’s scores/weights c have over just using the backbone model’s output z. They remain unclear how the Group Generator offered more insight than known types, and how the Group Selector gave more insight than the backbone model’s prediction.

Author response:
While voids and clusters are known, it has been unclear how much they contribute to omega and sigma for prediction. The research question here is how much they affect omega and sigma, and the weights from Group Selector offer some insights for that. Just using the backbone model’s output z is not faithful explanation of how this group contributes to the prediction.

Reviewer comment (Writing):
While SOP may have smaller group deletion and insertion error, it is more complex than a single attribution map and is essentially an ML model itself. This raises the question of how the user should actually use SOP. The case study gives indirect hints, but the practical usage remains unclear.

Author response:
One way to use SOP would be to look at top k groups affecting the prediction. We can also plot the histogram of different clusters of groups of groups that correspond to different a priori knowledge concept (such as voids and clusters) to obtain global explanations.

Reviewer comment (Evaluation):
Although the authors have done a good job on showing how their method offers improvements in insertion/deletion faithfulness, there is much less shown about the _quality_ of the feature groups/masks which are learned by SOP. The cosmology example is a good singular anecdote, but a more global analysis is needed to show that the interpretability of the feature groups is good. If the feature groups are very noisy and spread out all over an image (for example), then the method would not be very useful compared to another method which gives more contiguous or otherwise interpretable features. This is one of the core pieces that’s holding this paper back, in my opinion, and it would be great to see: 1) some examples of identified features (i.e. like saliency maps) for some examples of images, comparing SOP with other explanation methods; and 2) a global analysis quantifying the quality of the feature maps, comparing SOP with other explanation methods. It would be great to see if SOP is able to generate feature groups which are faithful but also interpretable (or at least, not significantly less interpretable than other methods).

Author response:
We have added more images for the groups generated and their comparisons with other explanations in Appendix D.3.

Reviewer comment (Theory):
The theoretical justification is certainly appreciated, but it does not appear entirely applicable to this work for a few reasons.

Author response:
Thank you for the suggestions. The theoretical justification is the motivation for creating grouped attributions. It is true that the example we provide is a toy example. We only show the case for when x=1. In real cases of images, we will have much more complicated interactions between features and thus we do not only have one value of x that is affected by not accounting for interactions between features.

Reviewer comment (Theory):
The paper assumes that the baseline image is the all-black image (all 0), but this is not a general assumption, especially in non-image datasets. Many of the Shapley-value works for interpretability are motivated (in part) by this observation. It should be possible to adjust the math and the implementation to account for non-zero baselines.

Author response:
Yes, it is simple to implement other image baselines. We focus on the faithful-by-construction framework, while can easily modify to achieve other image baselines.

Reviewer comment (Writing):
Equation 1: it would be good to formally define feature attribution $\alpha$; also, the powerset $\mathcal{P}$ is not used anywhere in the equation.

Author response:
Thank you for the suggestions and we will add formal definition of feature attribution and fix the typos and writings.

Reviewer comment (Writing):
Typo in Equation 8.

Author response:
Thank you for the suggestions and we will add formal definition of feature attribution and fix the typos and writings.

Reviewer comment (Theory):
Although SOP is being presented as a model-agnostic method, there is some reliance on the model architecture because of the need for a good embedding layer.

Author response:
For the embedding layer used in GroupGen in ImageNet and VOC, for patches, we are training a new simple convolutional layer that is the same stride size as its patch size, so the layer does not depend on the model. For Cosmogrid, because we are using segments instead of patches, we use the hidden state from the last layer of the model. So it contains more information than just a first embedding layer. If the original model already has reasonable performance on the end task, it is likely that the embeddings from the last layer are good. But it is true that there is some dependence on the model's internal hidden states for the embedding.

Reviewer comment (Theory):
In the convex program, what is the optimal $\alpha^{*}$ that is found? Because we are fixing $x = \mathbb{1}$ and $p(x)$, technically we already have a good idea of what the “right” attributions should be, and most configurations of attributions would be highly unreasonable (which general interpretability methods would not identify at all).

Author response:
Q1. For the monomial example, $\alpha\*$ is 1 for one group of $S_1 = \{1, …, d\}$ of all the features. For the binomial example, $\alpha\*_1 = 1$ for one group $S_1 \cup S_2$, and $\alpha\*_2 = 1$ for another group $S_2 \cup S_3$.

Reviewer comment (Theory):
There is certainly a connection to Shapley scores, which have the efficiency property; insertion and deletion error are closely related; having some background on this and explanation would be appreciated.

Author response:
Q2. Shapley values are computed by the number of times including a feature while perturbing all other features that still lead to the same output. They have the property that Shapley values for groups can be computed by summing the Shapley values for individual features. However, it is impractical to compute real Shapley values for images as the space of all features in a 224x224 image with pixel values in 3 channels from 0 to 255 is too large. SHAP computes an approximate by doing Monte Carlo sampling. However, SHAP score is not the same as Shapley values. Also, although summing all the shapley values will lead to the same prediction as the original prediction, this is not the same decision process as the original model and thus not faithful.

Reviewer comment (Novelty):
There may also be a connection to Novello, et al., 2022 (Making Sense of Dependence: Efficient Black-box Explanations Using Dependence Measure), which identifies feature importances based on masks as well, although feature groups is not a core focus there.

Author response:
Q3.Thank you for pointing out another related work. [Novello, et. al., 2022] assign scores to patches based on the kernel embedding of the distribution of each patch and the output y. As you also pointed out, they focus on individual feature importance, whereas we focus on importance of grouped features.

Reviewer comment (Experiments):
How does one choose the number of groups G? Once G is selected, are all of the G feature groups identified by SOP meaningful? Is there anything that prevents feature groups from being redundant? How can one select the most meaningful feature groups out of the G?

Author response:
Q4.​​ Currently the candidate number of groups G is number of features x number of heads (chosen). The sparse multihead attention then choose a subset of it. It is true that some feature groups are redundant. If the two feature groups are exactly the same, then the predictions and scores are also the same. When we interpret the final results, we combine the groups that are equivalent. It is also possible at test time to add an additional process to remove the redundant groups before going through the backbone model. GroupSelect assigns a score to each group for each class. The feature group that contributes the most to the prediction of a class is the one with the highest score of that class.

Reviewer comment (Theory):
More on the problem of redundancy: for a single input example, the same feature can be present in multiple masks/subsets; does this pose an issue for interpretability in this framework?

Author response:
Q5. Each mask/subset captures a different interaction within the group. Since the prediction does not depend on other features outside the group, it can still be interpreted as how this group of features contributes (without need for other features). If all the features are selected within a group, then it becomes uninterpretable. The groups generated by our group generator have around 60% non-zero features selected, so it should be interpretable.

Reviewer comment (Experiments):
Very Important: The experiments should have compared with existing methods that attempt to capture feature interactions, including (1) the cited Tsang et al 2020, (2) prior work on integrated gradients with feature attribution (https://aclanthology.org/2021.acl-long.71/), and (3) non-parametric search methods for insertion/deletion metrics (https://arxiv.org/abs/2106.00786). It has been known for a while that local linear approximations don’t capture feature interactions, and several methods have been proposed for resolving this.

Author response:
Thank you for pointing out the missing methods to compare. We havewill added experiments with Archipelago (Tsang et al 2020) for ImageNet and put in the revised Table 1. Due to time constraint, we have not compared with IDG and PLS as they only have code on text and IDG relies on a parser which is not available for images.

Reviewer comment (Evaluation):
Very Important: The primary quantitative metrics in the paper are not presented with sufficient detail, and this leads to the results being difficult to interpret. Is a .02 improvement of SOP over SHAP good or bad? It sounds very small. Sometimes the improvements are smaller or SHAP outperforms SOP.

Author response:
Our goal is to have guaranteed faithfulness which none of the other approaches have. The purpose of the metrics is to be consistent with the literature, but none of these metrics can perfectly capture faithfulness. The fact that our approach can guarantee faithfulness while matching SHAP performance is a good thing.

Reviewer comment (Evaluation):
Important: While I like the case study, the conclusions are somewhat mixed. (1) It is said that the method helps cosmologists see that voids are more important for predicting the omega constant than sigma, because the feature attribution is on average 55.4% for omega vs 54% for sigma. Surely this difference is so slight that we cannot conclude much from it. (2) The method’s explanations align with prior domain-specific knowledge (that’s good). (3) It’s said that explanations agree with conclusions from a 2020 study using gradient-based salience, and that it is “important that we find consistent results with our attention-based wrapper.” Why is it important to agree with a fundamentally flawed method from a previous study?

Author response:
We acknowledge a mistake on our part and thank the reviewer for their careful reading. Indeed we have confirmed with the cosmologists and this was a mistake in the text—the comparison between Omega and Sigma for voids was intended to show that there was no difference (54% to 55%), in contrast to the Omega and Sigma for clusters where there is a significant difference (14.8% to 8.8%). We have updated this in the revised manuscript. 

The previous gradient-based method does not give guarantee that the weights reflect how much the features contribute to the final prediction. Since gradient saliency methods are known to be unfaithful [Gilmer et al. 2020], this cast doubt into the cosmology community as for whether these results were true.

In fact, as we state in the third finding in Section 5, our findings are consistent with previous work's finding using gradient-based saliency maps. This can show that our findings are consistent with the findings of existing feature attributions in weak lensing maps, while we give more guarantees in how the weights faithfully reflect how much the groups are used by the model and validate the previous results.

It is important that we can validate their results (so we know that their results are correct), so we know for sure that voids can be more important than clusters for predicting these cosmological parameters. This is not to say that our method has to find the consistent results with fundamentally limited feature attribution methods. It would have been possible that we find opposite results.

Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2020). Sanity Checks for Saliency Maps. arXiv. https://arxiv.org/abs/1810.03292

Reviewer comment (Novelty):
Of Some Importance: I just want to note that I do not think the theoretical analysis adds much to the paper in its current state. We have seen strong impossibility theorems for rich model classes in the past year (e.g. the cited Bilodeau et al 2022 paper). A result showing that a linear model is not a good approximatation of a very nonlinear model, especially in high dimensions, is not very novel.

Author response:
While our theorems and that of Bilodeau et al 2022 both present impossibility results for feature attributions, we kindly point out that posed characterization of our results is incorrect, both on the assumptions and the resulting theorem. 

Bilodeau et al. 2022 put forth a result that says (put simply) that linear models cannot accurately capture complex models, where complexity is measured by having a large number of piece-wise linear components. Indeed, we agree that **if** we had shown that a linear model is not a good approximation of a highly non-linear model, then this would not be a novel contribution. This is also an unsurprising result (it is not surprising that a linear model cannot approximate a *highly non-linear* model). 

However, our result paints a significantly bleaker picture: we show that a linear feature attribution is unable to model the extremely **simple** functions in our theorems. Our examples distill the problem to the fundamental issue in its purest form: correlated features. Specifically, we show feature attribution is impossible with only **one** group of correlated features. This is the **polar opposite** assumption than that of Bilodeau et al. 2022: our theorem shows that standard feature attributions cannot hope to explain even simple functions if they contain just a single group of correlated features, which is more surprising than being unable to explain complex neural networks functions (and subsumes that case). 

Second, we provide not only a negative impossibility result for standard feature attributions, but also a positive result for grouped feature attributions that provides a path forward and motivates the approach in our submission. This is in contrast to Bilodeau et al. 2022, which only presents negative impossibility results standard feature attributions  without clear suggestions on where to go. 

In summary, our theoretical results differ in 
Assumption (we assume simple functions with a single correlation whereas Bilodeau et al. 2022 assume complex functions with many piece-wise linearities)
Theoretical results (we show both positive and negative results, whereas Bilodeau et al. only show negative results)
With this extended discussion, we hope the reviewer can better understand the theoretical results from both our paper and from Bilodeau et al. 2022. We have also included this discussion in the revision in Appendix B.1.

Reviewer comment (Writing):
The writing can be improved a lot. It took me a long time to figure out whether the paper is proposing an interpretation technique or a model fitting technique. A simple sentence in the abstract, "we modify an existing model to become interpretable, by using its predictions over a subset of features and then aggregating them", will be useful to readers.

Author response:
W1. Thank you for the advice on writing! We had a similar sentence in the abstract, but have used your suggestion to make this more clear in the updated revision.
> Specifically, we modify black-box models to have faithful interpretations, by aggregating predictions over varying subsets of features.

Reviewer comment (Evaluation):
While inspiring, the cosmology evaluation feels incomplete. Without a comparison control set (where cosmologists used LIME or some other method), it is not clear whether the benefits from SOP are unique. Also, there is a subtle problem in the conclusion of results: the results only show that _one_ predictive model tends to weigh voids higher than clusters. It does not say anything about the true process. It is possible that there is another model with same accuracy that weights voids and clusters differently. So I'm not sure what is the goal of the cosmology case study--is it to explain a ML model, or is it to understand the DGP/true model?

Author response:
W3. You bring up a great point: it is indeed possible that there is another model with the same accuracy that weights voids and clusters differently. However, in cosmology, we understand so little about the dynamics of dark matter and density fluctuations that understanding how *any* model (that achieves good performance) is weighting voids and clusters is an interesting problem. If there are multiple models that can use different patterns to achieve the same accuracy, then both of these are useful findings for cosmology. The baseline-study from [Matilla et al. 2020] is what our case study builds off from, which uses gradient saliency maps. However, the critiques of gradient saliency maps and their unfaithfulness [Gilmer et al. 2020, Sundararajan et al. 2017] led some of the cosmology community to have doubts on these initial results. Therefore the goal of our case study is twofold: (a)  to confirm whether these saliency-based results are consistent with faithful explanations (finding #3) (b) to help cosmologists learn more about the cosmos (findings #1 + #2). 


Matilla, José & Sharma, Manasi & Hsu, Daniel & Haiman, Zoltán. (2020). Interpreting deep learning models for weak lensing. Physical Review D. 102. 10.1103/PhysRevD.102.123506. 


Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2020). Sanity Checks for Saliency Maps. arXiv. https://arxiv.org/abs/1810.03292


Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017

Reviewer comment (Reproducibility):
It is not clear whether the sum-of-parts model requires any weight updates. Will it be better if aggregation parameter C is learnt? Is the backbone model same as an existing model to be explained, or is it a new model? Is the SOP model trained end-to-end or only the backbone is trained and SOP weightes can be inferred?

Author response:
Q1. The aggregation parameter C is learnt. We initialize it from an existing model but it is still being updated. The backbone model is an existing trained model that is completely frozen and not updated—this is typically a traditional deep learning model that has been trained on the same task. This backbone is used within GroupGen and GroupSelect to create the SOP model. The SOP model has additional trainable parameters (the attention weights in GroupGen and GroupSelect) which are trained end-to-end with the backbone model frozen.

Reviewer comment (Reproducibility):
Can you take a concrete example (e.g., ImageNet) and describe to readers how exactly you will create the SoP model?

Author response:
Q2. For ImageNet, we take a trained Vision Transformer image classification model from HuggingFace as the backbone model. You can imagine this backbone is then sandwiched between the GroupGen and GroupSelect modules to create an SOP model. Specifically: 
The GroupGen module uses an attention layer to select the groups from the 16x16 patches in an 224x224 image. 
Then, each group is passed separately into the backbone ViT model as the original image with patches not in the group masked out. 
The GroupSelect module then uses an attention layer to select a subset of groups to use to make a prediction. This attention layer uses the last hidden states from these group's ViT model outputs as keys and values, while the query is a learnable parameter initialized with the class weights.  These attention weights are then used to aggregate  the predicted logits from each group for each class.

Reviewer comment (Experiments):
Can you recreate Figure 5 using LIME or an existing feature attribution method on a standard non-SOP model? How different are the results?

Author response:
Q3. To evaluate how a structure is weighted in making the prediction, we would need to first have the groups of features and then check if they correspond to voids and clusters. For standard feature attribution methods like LIME, they attribute to individual features, and thus do not have a natural group created along the process. We could potentially do it with thresholded LIME. There is still the issue that since the weights from LIME are not faithfully representing how much the model is using these groups of features, even if we make a figure like Figure 5 using LIME, we do not know how to interpret the results.


In fact, as we state in the third finding in Section 5, our findings are consistent with previous work's finding using gradient-based saliency maps. This can show that our findings are consistent with the findings of existing feature attributions in weak lensing maps, while we give more guarantees in how the weights faithfully reflect how much the groups are used by the model and validate the previous results.

Reviewer comment (Evaluation):
Regarding Thm 1 and Thm 2, can you comment on counterfactual explanations? Since they are designed for deletion (but can also be defined for insertion), would they work better for Thm 1 and Thm 2? Would the feature attributions from CF explanations be better than other attributions? (for a discussion on CF explanations versus feature attributions, this paper may be helpful: https://dl.acm.org/doi/10.1145/3461702.3462597)

Author response:
Q4. Counterfactual explanations do not inherently give a score for different groups of features. The paper by [Mothilal et al. 2021] you pointed out does give a way to obtain feature attributions from counterfactual explanations. The feature attribution score for one feature is averaged over multiple counterfactual explanations. Thus, it is still giving standard individual feature attribution which has limitations over grouped attributions.

Reviewer comment (Presentation):
I found Table 1 very confusing. Certain numbers are bolded and italicized without any explanation as to why (it isn't bolding the best scores).

Author response:
We thank the reviewer for the constructive review. We want to apologize for the confusion on Table one. There was indeed a mistake with the bolding: the deletion score for integrated gradients should be bolded instead of italicized. This is corrected in the revised version. We originally intended the italicized numbers to indicate post-hoc methods, but acknowledge that in hindsight this was more confusing. We have cleaned this up in the revised version.

Reviewer comment (Writing):
I wish more attention was given to the Group Generator section.

Author response:
Q1: In the revision, we’ve updated the appendix (Appendix C) to contain more details on the group generated. We summarize this information for your convenience. 

For Group Generator, we first use either a patch segmenter (for ImageNet and VOC) or a contour segmenter (for Cosmogrid) to divide the image into segments. For patches, we separate them into 16x16 patches, and use a simple convolutional layer with patch size 16 and stride size 16 to convert each patch into an embedding of the patch. For contours, we use watershed segmenter in Scikit-learn, and use the pooler output (last hidden state of the [CLS] token) of each segment (masking out everywhere that is not the segment) as the embedding of the segment.

Then, we use the multiheaded self-attention described in the paper to assign scores to each patch. For all the segments, we have attention to all other segments. For example, for an 224x224 image in ImageNet, we have 196 patches, so when we use 2 heads in the multiheaded attention, we have potentially 196x2 group candidates. In order to reduce computation, we randomly dropout others and select only 20 groups to use during training, and also cap it at 50 for testing.

We use sparsemax which projects softmax scores to a simplex so that some scores are zeroed out completely. We use this so that each group does not contain all the segments in the image.

Reviewer comment (Writing):
Please elaborate on the Group Generator portion.

Author response:
Q1: In the revision, we’ve updated the appendix (Appendix C) to contain more details on the group generated. We summarize this information for your convenience. 

For Group Generator, we first use either a patch segmenter (for ImageNet and VOC) or a contour segmenter (for Cosmogrid) to divide the image into segments. For patches, we separate them into 16x16 patches, and use a simple convolutional layer with patch size 16 and stride size 16 to convert each patch into an embedding of the patch. For contours, we use watershed segmenter in Scikit-learn, and use the pooler output (last hidden state of the [CLS] token) of each segment (masking out everywhere that is not the segment) as the embedding of the segment.

Then, we use the multiheaded self-attention described in the paper to assign scores to each patch. For all the segments, we have attention to all other segments. For example, for an 224x224 image in ImageNet, we have 196 patches, so when we use 2 heads in the multiheaded attention, we have potentially 196x2 group candidates. In order to reduce computation, we randomly dropout others and select only 20 groups to use during training, and also cap it at 50 for testing.

We use sparsemax which projects softmax scores to a simplex so that some scores are zeroed out completely. We use this so that each group does not contain all the segments in the image.