Paper: Exploring the cloud of feature interaction scores in a Rashomon set (ICLR-2024)

Reviewer comment (Presentation):
The overall presentations need to be improved. Many figures do not have the axes' labels.

Author response:
- Following your suggestions, we have added axes' labels in Figure 3 and Figure 4 in the main text.  The remaining in Appendix will be fixed before the finalization of the camera-ready version.

Reviewer comment (Experiments):
The experiments should be conducted on a broader range of datasets.

Author response:
- In response to your recommendations, we have included an additional real-world application in Appendix I.  For your convenience, we summarized the experiment here. 

  

- To illustrate the usage of FISC, we applied our method to MXenes, an early transition metal carbides with two-dimensional (2D) structures exhibiting metallic conductivity and hydrophilicity. These materials are denoted by $M_{n+1}X_{n}T_{x}$, with $M$ representing an early transition metal such as Sc or Ti, $X$ signifying either Carbon or Nitrogen and functional groups $T$ (e.g., O, F, OH) for surface termination. By intercalating different ions and molecules like Li$^{+}$  and K$^{+}$,  MXenes' electrochemical attributes such as voltage, induced charge and capacity can be modulated, serving as a potential battery material. 

  

- We used a dataset that contains MXenes intercalated with Li$^+$, Na$^+$, K$^+$, and Mg$^{2+}$ ions with properties, voltage, induced charge and capacity. We represented the dataset by their categories, for instance, category Z encoded by 0–3 to describe Li, Na, K, and Mg, respectively, aiming to discover the interactions between different categories. For each property, we trained an MLP  to serve as the reference model and applied our framework to visualize FISC.  The results reveal that FIS exhibits a broad range when category Z is involved in interactions during the prediction of induced charge. This aligns with our common knowledge, given that ions, which form the category Z, are the cause of induced charge, shown in Appendix I. These insightful results offer a more comprehensive understanding of feature interactions within the Rashomon set and potentially guide researchers in making informed decisions for future research.

Reviewer comment (Novelty):
Is your work the first one to propose to use the loss change to measure the strength of feature interactions?

Author response:
- That is a great question. We conducted an investigation and did not find an exact equivalent definition in the existing literature. There are several works with similar ideas. For example, H-statistics [2] measure the strength of pairwise interactions based on the concept of Partial Dependence (PD) and it shows the marginal effect by averaging model outputs, rather than explicit loss like our work. The ANOVA [1] test examines the corresponding p-value for each pair of features as an interaction measurement, which isn't a precise match for our approach. Shapley interaction value and Shapley Taylor Interaction Index [3] are based on the game theory and measure the change in the function value by the addition of features. To our best knowledge, there is no identical definition in the literature, and we can claim that our method is the first one to define the feature interaction range in the Rashomon set.

Reviewer comment (Evaluation):
From my perspective, feature interactions typically don't have a definitive "ground truth." Therefore, how can you prove the superiority of your Feature Interaction Score (FIS) compared to other baselines without directly comparing it to a ground truth?

Author response:
- Good point! We appreciate your question and agree that feature interactions typically don't have a definitive "ground truth".  

- **(FIS vs FIS)** In certain scenarios, we can explicitly interact features in a function so that we have a "ground truth", such as functions in Synthetic Validation, Section 4.1. These synthetic cases serve as valuable benchmarks for evaluating the accuracy of feature interaction detection methods and facilitate comparisons among different approaches. 

- **(FIS vs FISC)** In more general cases, especially in real-world applications that lack a readily available "ground truth" for feature interactions, current methods primarily focus on FIS derived from a single model. As previously noted, the absence of ground truth values in such applications increases the risks associated with relying on a single model. Therefore, we recommend exploring a model class that provides a FISC and their distributions. This approach, in contrast to conventional methods, offers richer insights and a more comprehensive understanding. 

- **(FISC vs FISC)** When comparing FISC derived from different searching methods, the analysis typically involves examining FISC statistics such as std, mean, min and max. How to evaluate these comparisons is another interesting direction for further research.

Reviewer comment (Presentation):
What is MCR in Table 3?

Author response:
- Thank you for pointing that out. Model class reliance (MCR) refers to the range of feature importance (main effect in our definition) values in the Rashomon set. We followed the name convention, as it aligns with the common practice in Rashomon set-based feature importance exploration. We have added the explanation of MCR in section 4.2.1.

Reviewer comment (Novelty):
I am not fully convinced by the motivation of this paper. i.e., why do we need to explain feature interactions in a model class?

Author response:
- Interactions play an important role in model interpretation. Present methods often calculate interactions based on a single well-trained model, yet our demonstration reveals that a well-trained predictive model may not accurately preserve the true feature interactions in Limitations of previous work, Section 1. This discrepancy implies that existing methods could potentially yield inaccurate outcomes. Consequently, our aim is to explore feature interactions within a model class, which greatly increases the chance to uncover the "true" feature interactions. Moreover, with an increasing number of researchers supporting the exploration of a set of equally good models, the feature importance within a model class has gained prominence [1,2,3,4]. Notably, there is a gap of previous work specifically addressing feature interactions in a model class, making it a valuable avenue for exploration.  
- Given the critical role of precise feature interpretation in various scientific research endeavors, the imperative to sidestep inaccurate explanations is paramount to mitigate potential severe consequences. For example, 
	-  Drug–target interaction (DTI) is vital in drug discovery [5,6]. Predicting the interactions between drugs and targets becomes crucial in the drug discovery task. Different features, such as Enhanced Amino Acid Composition (EAAC) and Position-Specific Scoring Matrix (PSSM), extracted from protein sequences are employed for DTI prediction. These features are input into models for learning and predicting DTI. As the interaction prediction from a single model might be inaccurate, the drug discovery process would lead to misleading outcomes.  
	- To show the interaction variation in real-world application, we have included another example in Appendix using MXenes, an early transition metal carbide known for its metallic conductivity and hydrophilicity. The distribution of FIS offers valuable insights for researchers, and it can guide researchers in making informed decisions, potentially saving valuable resources that might otherwise be expended on redundant experiments.

Reviewer comment (Novelty):
The novelty of Shapley value calculation is not well presented. The focus is totally on the Rashomon set side.

Author response:
- We appreciate the acknowledgement of the Shapley value's value, and we extend our gratitude to the reviewer for recognizing our focus on the Rashomon set. Our paper is dedicated to exploring the feature interaction scores within the Rashomon set, and the feature interactions might be defined differently in future research. Importantly, we first propose and define the problem that a well-trained predictive model might not accurately capture the true feature interactions. To prevent that, we recommend exploring feature interaction strengths in a model class of approximately equally accurate predictive models, making the Rashomon set our focus, rather than Shapley value.

Reviewer comment (Writing):
In the example where the paper considers roots of a quadractic equation, it is unclear what "the input variables a, b, and c exhibit different feature interactions in these two models" means. Are the "two models" the two roots? Further explanation is required, it is confusing what the readers are asked to infer.

Author response:
Yes, the “two models” refer to the models for the two roots and they are $f_1(a,b,c) = \frac{-b+\sqrt{b^2-4ac}}{2a}$ and $f_2(a,b,c)=\frac{-b-\sqrt{b^2-4ac}}{2a}$. Both of them are error-free models. Our goal is to demonstrate that these two models induce different feature interaction scores. This is obvious as there is a sign difference in the numerators. More intuitively, consider a special case where $c=0$; the models reduce to $f_1(a,b,c) = 0$ and $f_2(a,b,c)=\frac{-b}{a}$, leading to distinct feature interactions among $a$ and $b$. We have made it clearer in Section 1.

Reviewer comment (Writing):
Consider adding a sentence to connect eq (6) to (2), currently it is a sentence fragment.

Author response:
Following your suggestion, we have revised sentences in Section 3.1 to connect eq (6) to (2).

Reviewer comment (Writing):
\mathcal{M} is not defined in (7) (its first use).

Author response:
We removed Eq. 7 and referred to Appendix D.0.2.

Reviewer comment (Theory):
How is eq (6) connected to (7)? It seems that the write-up connecting these two was skipped.

Author response:
The middle steps are illustrated in Appendix D.0.2 due to the page limitation. We removed Eq. 7 to improve the flow of the paper and save space.

Reviewer comment (Writing):
What does the paper mean by "then inversely calculate any order of interaction by decomposing..." at the end of 3.2.1.

Author response:
Given the greedy algorithm, we aim to search the Rashomon set and find main effects for features $(\boldsymbol{m}\_i)\_{i=1}\^{p}$. In order to find interaction effects, e.g., $\{i,j\}$, that requires search masks $m_{i,j}$, which is time-consuming. Instead of searching again, we can utilise the explored mask $m_i$ and $m_j$ and append them together as $(m_i | m_j)$, which is equivalent to $m_{i,j}$ as feature $i$ and feature $j$ are two different features. By doing so, we can inversely calculate any order of interactions.

Reviewer comment (Writing):
Notation \mathbbm{1}{i=1}^{p} will be more appropriate than $(1)_{i=1}^{p}$ in Algorithm 1. But more importantly, it is unclear how m+ is different from m-. They seem like a vector of size p of ones, i.e. both are the same. If so, then what is the difference between m_i+ and m_i-?

Author response:
We thank the reviewer for the suggestions, and we have incorporated the recommended changes in the format of 1 in Algorithm 1. $m_{i-}$ and $m_{i+}$ are different vectors for the same feature. Initially, $m_{i-}$ and $m_{i+}$ are identical vectors of 1s, but their divergence occurs during the search process. The $m_{i+}$ begins to increase, while the $m_{i-}$ starts decreasing. This distinction arises because our objective is to increase the loss value, and both actions serve this purpose effectively.

Reviewer comment (Writing):
Algorithm 1 is somewhat difficult to follow. Since this is the key contribution, can the authors explain the role of various terms such as "learning rate", \phi_s etc. are? Editing the algorithm to have comments, or updating 3.2.2 to reflect the terms in Alg. 1, can help. For instance, I am still unclear how the computations are taking place over multiple models, and if this is being influenced by "learning rate" somehow.

Author response:
Thank you for pointing that out. We have updated section 3.2.2 to reflect the terms in Algorithm 1. As models in the Rashomon can be characterized by masks, we alternatively search masks instead of searching all models. So, each mask corresponds to a model in the Rashomon set. This process is illustrated in Section 3.2.2 and Alg. 1, where we search in the “feature” directions, e.g., $m_i$ for $i$-th feature and $m_j$ for $j$-th feature, to quantify the main effects. 

- During the search process, we set a learning rate to regulate the quantity of models generated. Higher learning rates lead to a reduced number of models. The searching process continues until the loss condition is not satisfied $\mathcal{M}\_{\boldsymbol{m}\_i} \notin \mathcal{F}$.  This condition is imposed on the loss difference, denoted as $\phi\_{i} = \mathbb{E}[L(\mathcal{M}\_{\boldsymbol{m}\_i}(\boldsymbol{X}), \boldsymbol{y})] - \mathbb{E}[L(f(\boldsymbol{X}), \boldsymbol{y})]$. Any mask $\boldsymbol{m}\_i$ during training meets $\mathcal{M}\_{\boldsymbol{m}\_i} \in \mathcal{F}$ and is one of the target models for feature $\boldsymbol{x}\_i$ in $\mathcal{R}(\epsilon, f\^{*}, \mathcal{F})$.

Reviewer comment (Presentation):
Which functions is depicted in Fig. 3? We see x_0, x_1, and x_2, but it is unclear what was the original functional form.

Author response:
The function depicted in Fig. 3 is $f(x) = \bigwedge (x; \{x\_0^*, x\_1^* \} \cup x'\_2 ) + \bigwedge (x; \{x\_i\^{*}\}\_{i=11}^{30}) +\sum_{j=1}^{40}x_j$ from Table 1 in Section 4.1.  To provide clarity on the context, we have included the function in the caption of Fig. 3.

Reviewer comment (Writing):
What does "*" refer to in computational time section?

Author response:
The asterisk (*) used here indicates multiplication, and we have revised the sentence to minimize potential confusion.

Reviewer comment (Presentation):
Fig. 4 would benefit from axis labels, and needs to be referred to in the write-up.

Author response:
Thank you. We have addressed it by incorporating axis labels to enhance clarity and this figure is referred to in the "Going beyond the literature", Section 4.2.1.

Reviewer comment (Presentation):
Halo plots -- the blue curves for a fixed \epsilon account for all \phi_{i,j} (in the 2D case), but what is the x and y axis supposed to denote? Do the negative and positive values on the y axis carry meaning? Why does the x axis not have any axis values?

Author response:
This is an excellent question. $x$ and $y$ axis are decomposed $\phi_{i,j}$ into $\phi\_{i}$ and $\phi_{j}$ directions. Remember that $\phi\_{i,j}(\mathcal{M}\_{\boldsymbol{m}\_{i, j}}, f)$ is a function of $m_i$, $m_j$ and an unknown decomposed function $g(.)$. The red circle can be seen as the expected (without interaction) value of $m_i$, $m_j$ in $g(m_i, m_j) = \phi_{i} + \phi_{j}$, while the blue circle is the empirical value (with interaction) of same $m_i$, $m_j$, but in function $\hat{g}(m_i, m_j) = \phi_{i,j}$.  Here we only use the 2-D space to represent it, as the $g(.)$ and $\hat{g}(.)$ can be calculated from empirical loss.  

- The positive and negative values on the y axis represent the searching direction only. For each $m_i$ and $m_j$, we have two searching directions, denoted as $m_{i+}$, $m_{i-}$, $m_{j+}$, $m_{j-}$, referring to the algorithm, and both actions increase loss values. Thus, we can derive $\phi_{i+} + \phi_{j+} = \phi_{i+} + \phi_{j-} = \phi_{i-} + \phi_{j+} = \phi_{i-} + \phi_{j-}$. We have added values in x axis in figures to clarify the representation.

Reviewer comment (Writing):
The term "MCR" is undefined.

Author response:
Thank you for pointing that out. Model class reliance (MCR) refers to the range of feature importance (main effect in our definition) values in the Rashomon set. We followed the name convention, as it aligns with the common practice in Rashomon set-based feature importance exploration. We have added the explanation of MCR in section 4.2.1.