Paper: Learning High-Order Relationships of Brain Regions (ICLR-2024)

Reviewer comment (Writing):
In Section 4.1, first line: $X = [X_1, X_2, \dots, X_N]$ should be $X = [X_1, X_2, \dots, X_N]^T$ (i.e., a transpose operator should be inserted).

Author response:
Thanks for pointing this out and sorry for the confusion caused by the notation. We have inserted the transpose operator and gone through all the notations to make them more consistent and clear.

Reviewer comment (Theory):
In Eq. (4), a transpose operator should be inserted after the square brackets.

Author response:
Thanks for pointing this out and sorry for the confusion caused by the notation. We have inserted the transpose operator and gone through all the notations to make them more consistent and clear.

Reviewer comment (Experiments):
The authors mentioned that "Due to the data scarcity, training on individual datasets would result in serious overfitting." Each individual dataset includes at least 1000 subjects but the model performance is still not ideal. Can the model be applied to a dataset with less samples? Most clinical datasets are relatively small with < 1000 subjects. Is it applicable to apply the model on datasets with fewer samples?

Author response:
While we do think training on a small dataset is an interesting question, we argue that both our model and the state-of-the-art one are unable to solve it, and finding out the solution to such a problem is out of the scope.

To show it, we conducted experiments of training on individual datasets for both our model and BrainNetTF. The $r$ values:

|            | SST 1  | EN-back 1 | MID 1 | Rest 1 |
| ---------- | ------ | --------- | ----- | ------ |
| BrainNetTF | -0.029 | 0.091     | 0.075 | 0.081  |
| HyBRiD     | 0.103  | 0.085     | 0.002 | 0.060  |

The $P$-values corresponds to these $r$ values:

|            | SST 1 | EN-back 1 | MID 1 | Rest 1 |
| ---------- | ----- | --------- | ----- | ------ |
| BrainNetTF | 0.650 | 0.152     | 0.235 | 0.204  |
| HyBRiD     | 0.105 | 0.178     | 0.975 | 0.341  |

From this table, we can see all of the $P$-values> 0.05. Therefore, none of the existing models obtain **meaningful** results in this situation.

However, our work implies that if you have small fMRI datasets, you can usually **combine** them for a better result. We hope this could be a good solution to the small datasets issue.

Reviewer comment (Evaluation):
It would be helpful if the authors could discuss potential reasons why HyBRiD failed at Rest 1 dataset.

Author response:
Thanks for your suggestion. We inspect the performance on both the training, validation and test datasets. We summarize the performances of our model and the SOTA one in the table below

| model      | train | validation | test  |
| ---------- | ----- | ---------- | ----- |
| BrainNetTF | 0.967 | 0.484      | 0.334 |
| HyBRiD     | 0.984 | 0.491      | 0.223 |

We find our model outperforms BrainNetTF on both training and validation datasets but fails on test datasets. Therefore, given that Rest 1 is the **most noisy** dataset, we argue that the failure is because

1. Our model is more **expressive** than BrainNetTF, so it *overfits* the training dataset more than BrainNetTF.
2. The validation score is higher, indicating the failure reason is the **discrepancy** between the training dataset and the test dataset.

Additionally, Note that although HyBRiD is not the best on Rest 1 dataset, it is still the **second-best** one.

Reviewer comment (Evaluation):
Region importance. What about region importance for resting state data? If I understand it correctly, region importance is a metric for nodes. What about edges? Can you show how edges are connected under different conditions?

Author response:
We did not visualize it in the original submission because different from task states, where specific brain regions are activated in response to particular tasks, brain activities during resting states, are not driven by external tasks, leading to more **diffuse and less predictable** patterns of activation. This makes it harder to interpret. However, we appreciate the reviewer’s interest and we have included the visualizations and interpretations of *resting-state region importance* and *individual hyperedges* in **Appendix G** in the updated version. We would like to do more analysis and interpretations in the future.

Reviewer comment (Writing):
The notations in Section 5.1 Metric are not very clear to me.

Author response:
i) I agree that CPM(E, Y) is more accurate and Eq. 13 has been updated.

ii) $E$ denotes traditional pairwise edge weights, and it is a vector of length $K_p$, which is the total number of pairwise edges. We made it clearer by changing the notation $E$ to $\boldsymbol{W}_p$.

iii) We made it clearer by changing $\boldsymbol{w}$ to  $\boldsymbol{w}_h$ for Eq.13. And made $\boldsymbol{w}_h = \boldsymbol{w}$ as a special case when evaluating on our model. 

You can check Eq.12 and Eq.13 in the updated version.

Reviewer comment (Theory):
The input of CPM should include the prediction target Y, right? Maybe use CPM(E, Y)?

Author response:
i) I agree that CPM(E, Y) is more accurate and Eq. 13 has been updated.

ii) $E$ denotes traditional pairwise edge weights, and it is a vector of length $K_p$, which is the total number of pairwise edges. We made it clearer by changing the notation $E$ to $\boldsymbol{W}_p$.

iii) We made it clearer by changing $\boldsymbol{w}$ to  $\boldsymbol{w}_h$ for Eq.13. And made $\boldsymbol{w}_h = \boldsymbol{w}$ as a special case when evaluating on our model. 

You can check Eq.12 and Eq.13 in the updated version.

Reviewer comment (Writing):
What is the dimension of E? Is E equivalent to H+$\mathbf{w}$ in HyBRiD?

Author response:
i) I agree that CPM(E, Y) is more accurate and Eq. 13 has been updated.

ii) $E$ denotes traditional pairwise edge weights, and it is a vector of length $K_p$, which is the total number of pairwise edges. We made it clearer by changing the notation $E$ to $\boldsymbol{W}_p$.

iii) We made it clearer by changing $\boldsymbol{w}$ to  $\boldsymbol{w}_h$ for Eq.13. And made $\boldsymbol{w}_h = \boldsymbol{w}$ as a special case when evaluating on our model. 

You can check Eq.12 and Eq.13 in the updated version.

Reviewer comment (Writing):
CPM is evaluated on each model separately, right? If so, I think Eq. 13 should be defined separately for HyBRiD.

Author response:
i) I agree that CPM(E, Y) is more accurate and Eq. 13 has been updated.

ii) $E$ denotes traditional pairwise edge weights, and it is a vector of length $K_p$, which is the total number of pairwise edges. We made it clearer by changing the notation $E$ to $\boldsymbol{W}_p$.

iii) We made it clearer by changing $\boldsymbol{w}$ to  $\boldsymbol{w}_h$ for Eq.13. And made $\boldsymbol{w}_h = \boldsymbol{w}$ as a special case when evaluating on our model. 

You can check Eq.12 and Eq.13 in the updated version.

Reviewer comment (Writing):
Typos: Section 2.1 "Inupt" should be "Input"; Section 5.2 "conducte" should be "conduct"; Figure 7: "grpahical" should be "graphical".

Author response:
i) Thanks for pointing out. Typo fixed.

ii) Thanks for the suggestion. We acknowledge the importance of defining 'CPM' upon its first appearance in the text. However, due to the constraints of the abstract and introduction sections, providing a full definition is challenging. We have, therefore, incorporated a concise explanation, stating that CPM is the evaluation protocol for our method, which gives readers an adequate initial background. In addition, we also refer to the appendix. We have updated:

In abstract

> Our model outperforms the state-of-the-art predictive model by an average of 12.1%, regarding the quality of hyperedges measured by CPM, a standard protocol for studying brain connections.

In introduction

> We quantitatively evaluate our approach by a commonly used protocol
>
>
> for studying brain connections, CPM (Shen et al., 2017) (Appendix A), and show that…

iii) Sorry for the confusion. We have modified it to

> the resting state where the brain is not engaged in any activity (Rest)

iv) Thanks for pointing out. Typo fixed.

Reviewer comment (Writing):
CPM should be defined in the abstract and introduction upon its first occurrence.

Author response:
i) Thanks for pointing out. Typo fixed.

ii) Thanks for the suggestion. We acknowledge the importance of defining 'CPM' upon its first appearance in the text. However, due to the constraints of the abstract and introduction sections, providing a full definition is challenging. We have, therefore, incorporated a concise explanation, stating that CPM is the evaluation protocol for our method, which gives readers an adequate initial background. In addition, we also refer to the appendix. We have updated:

In abstract

> Our model outperforms the state-of-the-art predictive model by an average of 12.1%, regarding the quality of hyperedges measured by CPM, a standard protocol for studying brain connections.

In introduction

> We quantitatively evaluate our approach by a commonly used protocol
>
>
> for studying brain connections, CPM (Shen et al., 2017) (Appendix A), and show that…

iii) Sorry for the confusion. We have modified it to

> the resting state where the brain is not engaged in any activity (Rest)

iv) Thanks for pointing out. Typo fixed.

Reviewer comment (Writing):
In Section 5.1 Dataset, RS (resting state) should be defined.

Author response:
i) Thanks for pointing out. Typo fixed.

ii) Thanks for the suggestion. We acknowledge the importance of defining 'CPM' upon its first appearance in the text. However, due to the constraints of the abstract and introduction sections, providing a full definition is challenging. We have, therefore, incorporated a concise explanation, stating that CPM is the evaluation protocol for our method, which gives readers an adequate initial background. In addition, we also refer to the appendix. We have updated:

In abstract

> Our model outperforms the state-of-the-art predictive model by an average of 12.1%, regarding the quality of hyperedges measured by CPM, a standard protocol for studying brain connections.

In introduction

> We quantitatively evaluate our approach by a commonly used protocol
>
>
> for studying brain connections, CPM (Shen et al., 2017) (Appendix A), and show that…

iii) Sorry for the confusion. We have modified it to

> the resting state where the brain is not engaged in any activity (Rest)

iv) Thanks for pointing out. Typo fixed.

Reviewer comment (Evaluation):
Does the model generalize well across conditions or out-of-sample data? For example, if a model is trained on resting state data, can it be applied to predict task data?

Author response:
Thanks for your interest in the model's generalizability. In our experience, training on resting-state and generalizing it on task-state is difficult because resting-state data is more **noisy**. However, we do think it is a good idea so instead, we train our model on task-state data and generalize it one resting-state data.

|        | Training on all states, predicting on resting-state | Training on task-states, predicting on resting-state |
| ------ | --------------------------------------------------- | ---------------------------------------------------- |
| Rest 1 | 0.223                                               | 0.253                                                |
| Rest 2 | 0.730                                               | 0.475                                                |

The tables illustrate that our model demonstrates **a certain level of generalization**. For Rest 1, it is even better than training on Rest 1. We argue that it is because the model easily overfits on Rest 1 dataset, which is consistent with our response for **RC2.**

Reviewer comment (Novelty):
**The positioning of the paper is a problem.** The assertion that it captures high-order interaction is not substantiated, even though the feature selection model's entire motivation hinges on this claim. Certainly, the title emphasizes high-order interaction. However, the exact type of high-order interaction that the proposed model captures remains ambiguous. I would suggest considering the following papers, which were mistakenly overlooked. These papers seek to formally define what is being captured before attempting to estimate the interactions:  
- Rosas FE, Mediano PA, Gastpar M, Jensen HJ. [Quantifying high-order interdependencies via multivariate extensions of the mutual information](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.100.032305). Physical Review E. 2019 Sep 13;100(3):032305.  
- Varley TF, Pope M, Faskowitz J, Sporns O. [Multivariate information theory uncovers synergistic subsystems of the human cerebral cortex](https://www.nature.com/articles/s42003-023-04843-w). Communications biology. 2023 Apr 24;6(1):451.  
- Santoro A, Battiston F, Petri G, Amico E. [Higher-order organization of multivariate time series](https://www.nature.com/articles/s41567-022-01852-0). Nature Physics. 2023 Feb;19(2):221-9.

Author response:
We would like to politely point out that the high-order relations have been **intuitively** defined in the second paragraph of the Introduction (maximally informative and minimally redundant, MIMR) and **formally** defined in Equation 10 (IB principle). 

Specifically, our high-order relations are those that are most **predictive**, yet contain the **least** redundant information towards cognition. We have made the definition more clear and explicit in the updated version.

The definition of ours is different from that in the three papers, where they consider nodes in a high-order relation should be collectively correlated in terms of co-fluctuations or some metrics like O-information.

However, we appreciate the references you provided. We have included them and discussed the difference between their methods and ours in **Related Work (high-order relationships in fMRI)** in the updated version: Some of them are not scalable to high-order relations of a large degree, while some of them are inconsistent with our MIMR objective.

Reviewer comment (Theory):
**The clarity of the writing, with regard to the implementation**, is also inadequate in other sections. If we are estimating a hypergraph, then the edges, or node clusters, should form a cover rather than a partition. However, the regularization of the Mean Squared Error (MSE) used in Equation 11, as well as a preceding statement, both confirm the need for the edges to be disjoint, thereby suggesting a partition. This leads us back to the issue of positioning as it means that what is proposed is a clustering algorithm or the task of finding a partition. It's worth noting that node partitioning can still be conducted to recover high-order interactions, as has been exemplified in the neural imaging context, for instance, here:  
- Plis SM, Sui J, Lane T, Roy S, Clark VP, Potluru VK, Huster RJ, Michael A, Sponheim SR, Weisend MP, Calhoun VD. [High-order interactions observed in multi-task intrinsic networks are dominant indicators of aberrant brain function in schizophrenia.](https://www.sciencedirect.com/science/article/pii/S1053811913007970) NeuroImage. 2014 Nov 15;102:35-48.

Author response:
We would like to politely point out that the claim is incorrect. Our method indeed learns a **cover**, rather than a partition.

We guess the reviewer obtained such a conclusion because the statement in the submission

> The equality holds if and only if nodes are independent and hyperedges do not overlap

However, this is a fundamental misunderstanding of our **optimization objective**. This sentence just stated the condition in which equality holds in this inequality.

Specifically, Eq.10 shows

$$
I(H; X) \leq I_{\text{upper}}(H;X)
$$

where $I_\text{upper}(H;X)$ corresponds the right-hand-side of Eq.10. We optimize $I_\text{upper}(H;X)$ as a surrogate objective of  $I(H;X)$. In other words, we minimize $I(H;X)$ through minimizing $I_\text{upper}(H;X)$. This is a *common technique* in an abundance of works (e.g. in ELBO [1][2], in IB [3][4], and others [5][6]). Hence, the objective does not indicate partition.

On the contrary, if one wants to force the partition, he/she should optimize the **tightness** instead

$$
T(H, X) = I_{\text{upper}}(H;X) - I(H; X)
$$

where $T(H,X)$ denotes the tightness and it is minimized when hyperedges do not overlap. And minimizing the tightness is not what we are doing in this work.

Furthermore, the MSE term is **not for regularization**, it is the lower bound of informativeness in Eq.9, where $Y$ is the cognition score (defined in Section 2), which ensures the predictive performance of our high-order relations. The $I_{\text{upper}}$ actually corresponds to $\sum_{k=1}^K \sum_{i=1}^N \mathbb{H}[X_i](1-p_{\theta, i}^k)$ in Eq,10 & Eq.11, where one can find there is no force to partition at all.

The property that a node can be connected by **any number** (0 to $K$) of hyperedges is attributed to our masking mechanism. Each hyperedge is generated by a learnable mask independently, which is free to connect to the same node.

[1] Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. *arXiv preprint arXiv:1312.6114*.

[2] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. *Advances in neural information processing systems*, *33*, 6840-6851.

[3] Kim, J., Kim, M., Woo, D., & Kim, G. (2021). Drop-bottleneck: Learning discrete compressed representation for noise-robust exploration. *arXiv preprint arXiv:2103.12300*

[4] Wu, T., Ren, H., Li, P., & Leskovec, J. (2020). Graph information bottleneck. *Advances in Neural Information Processing Systems*, *33*, 20437-20448.

[5] Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y. (2018). Learning deep representations by mutual information estimation and maximization. *arXiv preprint arXiv:1808.06670*.

[6] Veličković, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., & Hjelm, R. D. (2018). Deep graph infomax. *arXiv preprint arXiv:1809.10341*.

Reviewer comment (Theory):
Overall, **the approach feels like an ad hoc method** for grouping the feature vectors via their predictive potential for a dependent variable. Even the input features are correlation coefficients. That is, the initial input matrix for each subject is the correlation matrix of which the goal is to subselect rows (or columns, which is equivalent due to symmetry) into K different groups.

Author response:
We would like to point out that our method is beyond feature grouping because 

1. A node is free to be connected by different hyperedges at the same time, which makes it different from grouping or clustering. 
2. Our method summarizes each hyperedge into a scalar, which is itself informative without the original features.
3. Our method also shows consistent predictive performance improvements other than selecting significant feature groups.
4. Our objective (MIMR) is based on a widely used principle and can be generalized on various tasks and various inputs.

Although in practice we choose the Pearson correlation as our input features, our method can generalize on other features (e.g. time signals). Our choice of Pearson correlation coefficients as input features is intended to enable a **direct comparison** with the state-of-the-art method [9], and they use Pearson correlations as their node features.

[9] Kan, Xuan, et al. "Brain network transformer." *Advances in Neural Information Processing Systems* 35 (2022): 25586-25599

Reviewer comment (Evaluation):
**Comparisons in Table 1 are highly problematic** as well. If the goal is to find high-order relations what does predictive quality of representations has to do with it? Table 1 in my opinion does not belong in a paper on high-order relations.

Author response:
This concern is a result of the misunderstanding of the model objective, as explained in our responses to Q1 and Q2, the high-order relations we want to discover in this work are the most **predictive** ones with minimal redundancy.

Reviewer comment (Experiments):
If the paper would be rewritten to focus on feature grouping and clustering, this approach may potentially work although not without changes. In this case, the proposed model needs to be compared with other approaches that do clustering or partition of random variables. For example, it seems appropriate to consider comparison with Deep Clustering.

Author response:
This concern is also a result of the misunderstanding of the model objective. Even if our task might be related to Deep Clustering, it differs a lot from the following aspects.

1. Our methods can learn a cover, which means a node can be connected to multiple hyperedges at the same time. However, most clustering methods usually assume each node only belongs to one cluster.
2. Most Deep Clustering methods usually need the supervision of cluster labels or semi-supervision signals of “must-links” and “cannot-links” [7].  However, we don’t have **such labels** besides the prediction labels. 
3. Deep Clustering methods don’t characterize the clusters with a **succinct summary**. However, our methods compute a weight for each hyperedge, which is **informative** toward cognition.
4. In contrast to clustering methods, which usually assign each node to a specific cluster, our approach selectively **prunes** nodes that are not crucial for cognition. For example, for the EN-back condition, only 64 out of 164 regions are connected by at least one hyperedge.

[7] Ren, Y., Pu, J., Yang, Z., Xu, J., Li, G., Pu, X., ... & He, L. (2022). Deep clustering: A comprehensive survey. *arXiv preprint arXiv:2210.04142*

Reviewer comment (Evaluation):
**Results are confusing.** They do not show individual "hyperedges" and analyze the ROIs that have grouped together and explain high-order interactions that grouped them. If the regularization enforces the partition, why are so many clusters overlap per my interpretation of Figures 5 and 4c?

Author response:
This is also a result of the misunderstanding in Q2. Our approach does not incorporate any regularization term to enforce the partition according to Eq.11 in the submission, so hyperedges are free to overlap with each other as long as the **overlap enhances the predictive performance**.

Reviewer comment (Writing):
The abstract states that CRM measures quality of hyperedges, however, CRM looks like a feature selection protocol that also has recommendations for assessing predictivity of the features. This is a disconnect between what is claimed and what is presented.

Author response:
We would like to politely point out that CPM is not a feature selection protocol. Feature selection is only **one of the steps** in the model. Here’s the justification why we use it as an evaluation protocol for our method:

   CPM was originally designed as a predictive model that predicts neurological outcomes based on edge weights. The evaluation of CPM could be measured by the correlation between predicted and ground-truth outcomes. The correlation is equal to the correlation between the sum of edges and the ground-truth outcome (The correlation is exactly the $r$ in Eq. 13) because CPM assumes **linear dependency** between the sum and the outcome. Hence, **one can expect a larger correlation if the edge weights are more correlated to the outcomes (and thus are more predictive)**.

Reviewer comment (Writing):
The beginning of the second paragraph of the Introduction needs references to support the claim. The last phrase of that paragraph needs a rewrite "of the intricate behind brain regions".

Author response:
(1) Thanks for the suggestion. References added.

   (2) Rewrite the sentence to 

   > hinders the accurate understanding of brain region interactions (Cao et al., 2022; Richards et al., 2019)

Reviewer comment (Writing):
Section 2.1 "Inupt" - Do you mean Input?

Author response:
Thanks for pointing out. Typo fixed

Reviewer comment (Writing):
Section 2.2 describes feature selection - why is this tied to high-order interactions. Confusing.

Author response:
Section 2.2 does not describe feature selection, it describes the background of the metric in Eq. 13, which uses CPM to evaluate the predictive performances of our learned hyperedges. For a more coherent presentation, we have moved all contents about CPM into Appendix B.

Reviewer comment (Writing):
Section 4.1 what do you mean by "N-dimensional shallow embedding layer parameterized by". It would be best if all operations and parameters were clearly defined. I assume this is a linear transformation but that is a guess.

Author response:
A shallow embedding layer, such as the token embedding layer commonly used in natural language processing (referred to in PyTorch as `torch.nn.Embedding`), is a simple lookup table that stores embeddings of a fixed dictionary and size. For better comprehension, we have omitted the term and instead provided a direct explanation:

   > …,  where $p^k_{\theta, i}, i=1,2,\cdots, N$ are learnable parameters.

Reviewer comment (Experiments):
Baselines: "standard" method mentioned there is unclear. What is standard? How is it defined?

Author response:
“standard” is defined in the **Baselines** paragraph in Section 5.1 in the original submission

   > standard method, which is exactly the classical pairwise-based one.

   we have added references for such methods and made the definition clearer in the paper.

   > standard method, which is exactly the classical standard method, which is exactly the classical method that predicts outcomes based on pairwise edges (Shen et al., 2017)

Reviewer comment (Presentation):
Consider fixing capitalization in your bib file. To do that, you can go over the cited papers in your .bib file and put all words you want to preserve capitalization of in additional curly braces. Like {fMRI}.

Author response:
Thanks for pointing out. Fixed.

Reviewer comment (Theory):
Formulating the hyperedge inference problem into a feature selection (MRMR-like) problem is interesting, yet not valid, at least in the context of functional connectivity analysis. Regions that are predictive together towards a certain cognitive condition do not imply that they are functionally connected.

Author response:
We believe that there is a misunderstanding on the motivation. Traditional methods [1][2] typically define functional connectivity based on **similarity or correlation** (like Pearson correlation, and mutual information). However, this kind of connectivity is **not informative** toward a neurological outcome (see the performance of standard pairwise baseline in Table 1 as evidence) and might fall short in providing insights relevant to the outcome. As a result, it is crucial to learn these connections taking the outcome into account like what has been done in [3][4]. Therefore, our work does not seek to identify the traditional connections.

Instead, as mentioned in 3rd paragraph of the submission, the objective of our method is to discover high-order connections that are **predictive** toward a cognition score (a part of our **MIMR objective**). And for regions that are connected by a predictive high-order relation, the relations between these regions can be highly non-linear and complex, and far beyond any existing manually-designed metrics (e.g. Pearson correlation, mutual information).

[1] Shen, X., Finn, E. S., Scheinost, D., Rosenberg, M. D., Chun, M. M., Papademetris, X., & Constable, R. T. (2017). Using connectome-based predictive modeling to predict individual behavior from brain connectivity. *nature protocols*, *12*(3), 506-518.

[2] Boyle, R., Connaughton, M., McGlinchey, E., Knight, S. P., De Looze, C., Carey, D., ... & Whelan, R. (2023). Connectome‐based predictive modeling of cognitive reserve using task‐based functional connectivity. *European Journal of Neuroscience*, *57*(3), 490-510.

[3] Kawahara, J., Brown, C. J., Miller, S. P., Booth, B. G., Chau, V., Grunau, R. E., ... & Hamarneh, G. (2017). BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment. *NeuroImage*, *146*, 1038-1049.

[4] Mahmood, U., Fu, Z., Calhoun, V. D., & Plis, S. (2021). A deep learning model for data-driven discovery of functional connectivity. *Algorithms*, *14*(3), 75.

Reviewer comment (Reproducibility):
It is recommended to discuss how the DimReduction MLP could be trained with a large number of nodes.

Author response:
We apply DimReduction MLP independently on each hyperedge. As specified in Eq. 6 in the submission, for a hyperedge we first average features of nodes in that hyperedge by ${\boldsymbol{m}^k}^T \boldsymbol{h}^k$ and obtain the hyperedge features in $\mathbb{R}^{d}$. After that, the DimReduction MLP compresses the dimensionality from $d$ to $1$ as a scalar weight for the hyperedge. The parameters of the DimReduction MLP are shared for every hyperedge and every dataset.

Reviewer comment (Reproducibility):
Are the linear-head Fl shared across hyperedges or trained separately for each hyperedge?

Author response:
As specified in Eq.7, the inputs of the linear-head $\mathcal{F}_l$ are scalar weights of all hyperedges. Consequently, the notion of shared vs. separate linear-head functions in terms of hyperedges does not apply in this context regarding hyperedges. However, $\mathcal{F}_l$ is shared across all datasets.