Paper: Learning From Simplicial Data Based on Random Walks and 1D Convolutions (ICLR-2024)

Reviewer comment (Experiments):
The impact of parameter choices could be studied more carefully.

Author response:
We added an ablation study to the paper that explores the influence of considered adjacency, the size of the local window, and the length of the random walks on the performance on the social contact datasets.

Reviewer comment (Experiments):
A discussion on the utility of a simplicial perspective might enrich the paper. As it stands now, the experiments deal with simplicial data. The scope of the paper could be improved substantially if the simplicial perspective was shown to be *crucial* for good performance. The reviewer considers this a minor weakness since the paper already makes a strong contribution under the assumption that simplicial complexes are the 'right' thing to model the data. Showing this empirically would just be an additional enhancement. The experiment on social contact networks is a good start for this, but a more in-depth discussion would be appreciated if possible.

Author response:
We agree that highlighting the importance a simplicial perspective on datasets would be benefitial.  
However due to space constraints we considered this outside the scope of our paper.  
The comparison between SCRaWl and CRaWl (Figure 7) provides an initial indication towards this topic and other authors considered this comparison as well, e.g. Bodnar et al, 2021.  

We argue that this topic should be dealt with in a more dedicated work and on a more abstract level not focused on one particular neural network architecture but rather with different graph neural networks and simplicial neural networks in mind.

Reviewer comment (Writing):
As a minor style issue, please use `\citep` and `\citet` (when using `natbib`) consistently. The former is meant for parenthetical citations, the latter for in-text citations.

Author response:
We added the suggested survey paper as reference for expressivity in context with WL and fixed the minor style issues.

Reviewer comment (Experiments):
How are the random walk parameters (length / number of walks) affecting performance? Could predictive performance be shown as a function of these parameters, or, even better, could theoretical guarantees be provided?

Author response:
We added an ablation study to the paper that explores the influence of considered adjacency, the size of the local window, and the length of the random walks on the performance on the social contact datasets.

Reviewer comment (Theory):
Would the proposed method also extend to *hypergraphs* or other combinatorial complexes? This seems likely, given that only a notion of a random walk is required.

Author response:
The approach should also work on combinatorial complexes, provided an analogous notion of a random walk is defined.  
This is similar to the case of cell complexes as explained in the conclusion.  
For hypergraphs, the notion of upper and lower adjacency is not well defined, which is why we did not consider them in our work.  
Of course, ignoring this (e.g., by merging the respective columns into one) would yield a valid approach, but we did not investigate the performance of such a method in detail here.

Reviewer comment (Experiments):
Given the performance considerations, are there any practical limitations in terms of data set size that could be given? This might be useful to assess the suitability for certain data sets in advance.

Author response:
For larger datasets, reducing the number of random walks is a viable option to reduce complexity.  
Choosing one random walk per simplex makes is very likely to cover each simplex sufficiently often as a center simplex in a local window, but this property is neither ensured nor necessary.  
It is also worth considering to use smaller parameters during training and larger parameters during inference.

Reviewer comment (Novelty):
The paper has missed some important related studies or detailed discussions about them. For example, there are several random walk-based methods [1, 2] in a simple graph that use a similar idea. While they have been briefly mentioned, there are important questions about them that are not discussed in the paper, such as whether there is any connection between SCRaWl and these methods, or if SCRaWl is the extended version of these studies to higher-order data. A very similar study on higher-order graphs is CATWALK [3], a hypergraph learning method that similarly uses higher-order random walks to learn higher-order patterns. Given that hypergraphs can be seen as a general case of simplicial complexes, their walk sampling is exactly the same as this work, which limits the novelty of this paper's idea and its model design.

Author response:
**References**

We incorporated the suggested references into our work.
References [1] and [2] have already been cited in our related work section before.
In particular, [2] is the primary work we built upon by extending their approach to simplicial complexes.
We believe this is made clear in the current version of the paper already, e.g. in the introduction: "[...] here we take inspiration from random walk-based graph neural networks on graphs (Tönshoff et al., 2023) and explore the use of a random walk-based learning architecture for simplicial complexes."

It is our understanding that [4] focuses on hyperedge link prediction and their proposed method is therefore not (directly) applicable to the tasks considered in our paper.
Similarly, [3] deals with temporal data, which is not the focus of our paper, though they also explain how their approach can be applied to node classification on static hypergraphs.
We however want to point out that this work was published after the submission deadline of our paper.

We added a paragraph to our related works section that mentions hypergraphs as another paradigm to represent higher-order interactions and reference to some noticeable hypergraph neural networks, including the proposed references [3] and [5].
We further extended the comparison on the social contact datasets with hypergraph neural networks (Table 3 in the appendix).

Reviewer comment (Experiments):
There is a lack of ablation study on the method architecture. It is not clear what the contribution of each component of SCRaWl is. Questions remain such as how using a simple random walk instead of a higher-order random walk can affect the performance, and how each of the six features contributes to the performance of SCRaWl.

Author response:
**Ablation Study**

We added an ablation study on the social contact datasets to the appendix where we compare the influence of the window size $s$, the walk length $\ell$, and of lower and upper adjacency when sampling the walks.

Reviewer comment (Novelty):
In the rebuttal phase, there remains a lack of discussion with very similar works (e.g., [2]). The existing discussion briefly mentions taking inspiration from random walk-based graph neural networks on graphs (Tönshoff et al., 2023), but this is not enough. The idea of using 1D Convolutions is previously discussed in [2], and the only remaining contribution in this paper is to design higher-order random walks on simplicial data. The proposed walk is also similar to that in [3], with the only difference being that the random walk in [3] is temporal, which can be simply adjusted to a single timestamp. The authors claim [3] was published after the submission deadline, but it has been available since June 2023 (3–4 months before the deadline).

Author response:
**References**

We incorporated the suggested references into our work.
References [1] and [2] have already been cited in our related work section before.
In particular, [2] is the primary work we built upon by extending their approach to simplicial complexes.
We believe this is made clear in the current version of the paper already, e.g. in the introduction: "[...] here we take inspiration from random walk-based graph neural networks on graphs (Tönshoff et al., 2023) and explore the use of a random walk-based learning architecture for simplicial complexes."

It is our understanding that [4] focuses on hyperedge link prediction and their proposed method is therefore not (directly) applicable to the tasks considered in our paper.
Similarly, [3] deals with temporal data, which is not the focus of our paper, though they also explain how their approach can be applied to node classification on static hypergraphs.
We however want to point out that this work was published after the submission deadline of our paper.

We added a paragraph to our related works section that mentions hypergraphs as another paradigm to represent higher-order interactions and reference to some noticeable hypergraph neural networks, including the proposed references [3] and [5].
We further extended the comparison on the social contact datasets with hypergraph neural networks (Table 3 in the appendix).

Reviewer comment (Experiments):
There is a lack of comparison with important baselines such as [3, 5]. Although results from [3] are reported, it is unclear if the experimental setting is the same as in this paper. Questions about whether hyperparameter tuning and preprocessing steps are consistent across methods remain. Reporting results from other works is considered unfair without consistent setups.

Author response:
**References**

We incorporated the suggested references into our work.
References [1] and [2] have already been cited in our related work section before.
In particular, [2] is the primary work we built upon by extending their approach to simplicial complexes.
We believe this is made clear in the current version of the paper already, e.g. in the introduction: "[...] here we take inspiration from random walk-based graph neural networks on graphs (Tönshoff et al., 2023) and explore the use of a random walk-based learning architecture for simplicial complexes."

It is our understanding that [4] focuses on hyperedge link prediction and their proposed method is therefore not (directly) applicable to the tasks considered in our paper.
Similarly, [3] deals with temporal data, which is not the focus of our paper, though they also explain how their approach can be applied to node classification on static hypergraphs.
We however want to point out that this work was published after the submission deadline of our paper.

We added a paragraph to our related works section that mentions hypergraphs as another paradigm to represent higher-order interactions and reference to some noticeable hypergraph neural networks, including the proposed references [3] and [5].
We further extended the comparison on the social contact datasets with hypergraph neural networks (Table 3 in the appendix).

Reviewer comment (Experiments):
There is still no ablation study, and initial questions remain unanswered: (i) How would using a simple random walk instead of a higher-order random walk affect performance? (ii) How does each feature contribute to SCRaWl's performance? (iii) What is the contribution of 1D Convolutions, and what would happen if a simple linear layer were used instead?

Author response:
**Ablation Study**

We added an ablation study on the social contact datasets to the appendix where we compare the influence of the window size $s$, the walk length $\ell$, and of lower and upper adjacency when sampling the walks.

Reviewer comment (Experiments):
The major concern I could see is memory and computation as it contains the matrices for k levels of simplices. And, each walk consists of six matrices.

Author response:
We agree that the memory footprint is considerable, especially for large datasets.
   However, this is generally the case for simplicial complexes as the simplicial property forces exponentially many sub-simplices for any higher order simplex.
   The feature matrices themselves can be computed on the fly and are very sparse in the later columns (containing the structural information about connectivity and identity).
   The first columns contain features that any reasonable approach needs to store as well.
   An advantage of our approach is, however, that we can vary the scalability, e.g., by selecting the window sizes and walk lengths, which is not possible for classical message passing architectures. Investigating such trade-offs in more detail (more broadly: how to optimally leverage higher-order information, while keeping computations tractable) is an interesting topic we intend to explore in future work in more detail.

Reviewer comment (Experiments):
Why are walks passed in every layer? Is it not enough to use it only at the input?

Author response:
The feature matrices change in every layer (because the hidden states are updated), thus the random walks are passed to every layer.

Reviewer comment (Presentation):
How are the output from k-simplices combined for the final output? I could see three different outputs in Fig. 3.

Author response:
This depends on the dataset at hand and the considered targets.
   For the semantic scholar dataset, the outputs $y_0, ..., y_5$ are compared as is with the targets.
   For the social contact dataset, only the outputs $y_0$ are considered.

Reviewer comment (Evaluation):
Fig 6. results are on par with MPSN. Justification of why it works differently on two different kinds of datasets would be much appreciated. Results are improved significantly for social networks although it.

Author response:
For the semantic scholar dataset, the results of SCRaWl and MPSN are close to perfect, so we cannot expect any strong improvement beyond MPSN.
   The social contact datasets show that SCRaWl is able to outperform MPSN.

Reviewer comment (Reproducibility):
Why are walks computed on the fly or they are just sampled on the fly? Can’t we precompute the walks which is just one time processing before training?

Author response:
Yes, the walks can be precomputed as well. This is actually also implemented in our code.
However, this does not have any influence on imputation accuracy, but only on training time, which is why we did not include it in the paper.
Notice that the same random walks should not be reused in every epoch to avoid overfitting to these specific walks.

Reviewer comment (Novelty):
Compared to this work, a recent work [1] seems to provide more principled insights into exactly the same topic. Can the authors discuss more of its edge over this work?

Author response:
We thank the reviewer for their feedback.

We would like to point out that the mentioned paper was published after the submission deadline of this paper.
However, we agree that the paper provides interesting results and discuss it now in our work.
Notice that the paper is similar to RWNN (already in our related work) in that the authors employ random walk probabilities as additional node and edge features, which enhance the main graph neural network.
Different to RWNN, they use random walks on simplicial complexes (e.g. by lifting).
In contrast, in our work, the concrete random walks are not used as enhancement, but as the main building block of the network.