Paper: SaNN: Simple Yet Powerful Simplicial-aware Neural Networks (ICLR-2024)

Reviewer comment (Writing):
Certain definitions regarding the types of operators and features are not laid out clearly enough, which leads to ambiguity in the paper on a technical level. As noted in the list of questions and suggestions, the claimed properties of the proposed models are not clearly true, possibly due to this misunderstanding.

Author response:
We sincerely appreciate your positive feedback and constructive remarks on our paper. We acknowledge that the unclear definitions may have led to some confusion regarding the properties of the proposed models. Upon reflection, we agree that the orientations of signals and simplices were not sufficiently defined, potentially leading to misunderstandings. We have made revisions to the paper, as detailed below, to enhance clarity and readability.

Reviewer comment (Theory):
There are ambiguities around orientation equivariance and the use of oriented operators built from the incidence matrices. These ambiguities are the main reason for suggesting the paper be rejected, and addressing this point would be the most important step to change the reviewer’s opinion.

Author response:
We appreciate your feedback and recognize that our initial description, which stated "The $(i, j)$th entry of $\\mathbf{B}_k$ is non-zero if the $i$th $(k−1)$-simplex is a boundary simplex of the $j$th $k$-simplex", could have been clearer. 

We have clarified in the revised manuscript that the $(i,j)$th entry of $\mathbf{B}_k$ can be either $+1$ or $-1$, depending on the relative orientations of the $i$th $(k−1)$-simplex and the $j$th $k$-simplex. Even though in the introduction we mentioned about orientations of higher-order simplices as “Higher-order simplices are usually oriented, ensuring a consistent node arrangement within each simplex, facilitating tasks like determining information flow directions along the edges.”, we did not formally define the orientations of features on the simplicial complex in relation to the reference node ordering. Therefore, we have now specified in the revised manuscript that signs of features on simplices are determined based on the reference node arrangement within each simplex. Additionally, when features are unoriented, we clarify that there is no need for a reference orientation of simplices. In such cases, we consider unoriented incidence matrices.

To this end, we have incorporated the following points in Section 2 (Background) at appropriate locations:

"Each simplex has an orientation defined by a standardized vertex order, typically ascending or descending, establishing a standard node arrangement within each simplex."

"The non-zero entries of an oriented incidence matrix $\mathbf{B}_k$ can be either $+1$ or $-1$, reflecting the relative orientations of the $i$th $(k−1)$-simplex and the $j$th $k$-simplex."

"The sign of feature $\mathbf{X}_k[\sigma_k]$ is determined based on the reference orientation of $\sigma_k$."

"For unoriented features, there is no need for a reference orientation of simplices. In such cases, we consider unoriented incidence matrices. Specifically, the $(i, j)$th entry of an unoriented incidence matrix  $\mathbf{B}_k$ is $1$ if the $i$th $(k−1)$-simplex is a boundary simplex of the $j$th $k$-simplex, and $0$ otherwise."

We believe that these revisions will enhance the comprehension of the orientation equivariance concept within our paper.

Reviewer comment (Theory):
There are some details missing regarding the type of data being handled. In particular, the incidence matrices are not defined in a way sufficient for the discussion in the paper. Normally, incidence matrices have values of ±1 depending on a chosen reference orientation (usually given by an ordering of the nodes). The signs of the features on the simplices are determined relative to the same reference orientation, giving meaning to the notion of orientation equivariance. Without discussing these aspects, orientation equivariance is not a meaningful concept in the paper’s context.

Author response:
We appreciate your feedback and recognize that our initial description, which stated "The $(i, j)$th entry of $\\mathbf{B}_k$ is non-zero if the $i$th $(k−1)$-simplex is a boundary simplex of the $j$th $k$-simplex", could have been clearer. 

We have clarified in the revised manuscript that the $(i,j)$th entry of $\mathbf{B}_k$ can be either $+1$ or $-1$, depending on the relative orientations of the $i$th $(k−1)$-simplex and the $j$th $k$-simplex. Even though in the introduction we mentioned about orientations of higher-order simplices as “Higher-order simplices are usually oriented, ensuring a consistent node arrangement within each simplex, facilitating tasks like determining information flow directions along the edges.”, we did not formally define the orientations of features on the simplicial complex in relation to the reference node ordering. Therefore, we have now specified in the revised manuscript that signs of features on simplices are determined based on the reference node arrangement within each simplex. Additionally, when features are unoriented, we clarify that there is no need for a reference orientation of simplices. In such cases, we consider unoriented incidence matrices.

To this end, we have incorporated the following points in Section 2 (Background) at appropriate locations:

"Each simplex has an orientation defined by a standardized vertex order, typically ascending or descending, establishing a standard node arrangement within each simplex."

"The non-zero entries of an oriented incidence matrix $\mathbf{B}_k$ can be either $+1$ or $-1$, reflecting the relative orientations of the $i$th $(k−1)$-simplex and the $j$th $k$-simplex."

"The sign of feature $\mathbf{X}_k[\sigma_k]$ is determined based on the reference orientation of $\sigma_k$."

"For unoriented features, there is no need for a reference orientation of simplices. In such cases, we consider unoriented incidence matrices. Specifically, the $(i, j)$th entry of an unoriented incidence matrix  $\mathbf{B}_k$ is $1$ if the $i$th $(k−1)$-simplex is a boundary simplex of the $j$th $k$-simplex, and $0$ otherwise."

We believe that these revisions will enhance the comprehension of the orientation equivariance concept within our paper.

Reviewer comment (Theory):
The validity of the example in Section 4.1 is questionable. The paper claims that all simplices are given a scalar feature value \(a\), yet the matrices acting on these features have an orientation. This seems to involve using an *oriented operator* on *unoriented features*. Property 1 in this example is therefore difficult to claim, since orientation equivariance describes *oriented operators* acting on *oriented features*, with the choice of initial orientation being irrelevant to computation.

Author response:
We appreciate your feedback regarding Section 4.1. In Section 4.1, we consider unoriented simplices. The details are as follows.
  
In Section 4.1, we explore example functions that adhere to the conditions in Theorem 4.2, demonstrating that SaNN is as powerful as the SWL test. It is important to note that the SWL test differentiates non-isomorphic simplicial complexes based on structure, not the features they carry as in the commonly used WL test. Consequently, it assumes uniform initial features (colors) across all simplices, which are then updated iteratively [Please refer to Section 3 in Bodnar et al., 2021].

Our goal in showing equivalence to the SWL test is to demonstrate that SaNN assigns distinct embeddings to two non-isomorphic simplicial complexes, irrespective of their features, if the SWL test assigns different colors to their structures. In this context, we consider unoriented uniform features across all simplices, thus dealing with unoriented simplicial complexes. 

However, SaNN is also capable of handling oriented features in practice. For instance, in trajectory prediction where flow has orientation, we consider oriented simplicial complexes. In such scenarios, we can also establish orientation equivariance as detailed in the paper, hence validating Property 1. In sum, the two settings are different as highlighted by the reviewer and we have clarified this in the paper. To enhance clarity, we have slightly revised the introductory text of Section 4.1 as follows:

"In this section, we discuss example functions that fulfill the conditions outlined in Theorem 4 and demonstrate that SaNN is as powerful as the SWL test. The SWL test [cf. Appendix C] distinguishes non-isomorphic simplicial complexes based on structure, assuming uniform initial features (colors) across all simplices. To establish equivalence with the SWL test, we consider simplicial complexes with a uniform scalar feature $a$ on all simplices, without attributing any orientation to the features or the simplices. However, it is worth noting that SaNN can also process oriented features in practice, and in such cases, we work with oriented simplicial complexes (or incidence matrices) as defined in Section 2."

Reviewer comment (Evaluation):
This issue also leads to an incorrect comparison for isomorphism testing, since the erroneous imposition of differently oriented features relative to chosen orientations could cause SaNN to produce a “false negative” — erroneously indicating that two isomorphic complexes are different.

Author response:
In response to your comment, we reemphasize that isomorphism testing primarily focuses on the structure of simplicial complexes, not the features they carry. The objective is that SaNN should assign distinct embeddings to two different structures that the SWL test identifies as non-isomorphic. When establishing equivalence to the SWL test, our reference to requiring injective functions for aggregation pertains to structure, not initial features. This means that we need distinct embeddings for two unique structures, assuming all features are identical. As we theoretically prove in the paper, under the conditions stated in Theorem 4.2, starting with uniform initial features (colors) across all simplices, SaNN is as powerful as the SWL test in distinguishing the structures of non-isomorphic simplicial complexes.

Reviewer comment (Theory):
In the **Insights** section of Section 5.1, the claim that “the superior performance of SaNN also proves the orientation equivariance of SaNN experimentally” is incorrect. Orientation equivariance is a straightforward mathematical property that does not guarantee good performance, nor does good performance guarantee orientation equivariance. While these properties may be related, the claim that one proves the other is not justified.

Author response:
Thank you for pointing it out. Indeed, you are correct. Orientation equivariance does not automatically equate to superior performance. Similarly, good performance on a specific dataset does not necessarily imply orientation equivariance. Our intention was to highlight that SaNN's good performance signifies its effective use of the orientations of flows for trajectory prediction. Therefore, we will revise the sentence as follows: 

"The good performance of SaNN also signifies its effective use of the orientations of flows for trajectory prediction."

Reviewer comment (Theory):
The claims in Section 4.2, while reasonable at first glance, are not explained in enough detail. Permutation equivariance is straightforward to establish, so it is not a major concern, but orientation equivariance suffers from the same problems outlined above and requires clarification on the type of simplicial features and relevant operators.

Author response:
We appreciate your feedback and recognize that our initial description, which stated "The $(i, j)$th entry of $\\mathbf{B}_k$ is non-zero if the $i$th $(k−1)$-simplex is a boundary simplex of the $j$th $k$-simplex", could have been clearer. 

We have clarified in the revised manuscript that the $(i,j)$th entry of $\mathbf{B}_k$ can be either $+1$ or $-1$, depending on the relative orientations of the $i$th $(k−1)$-simplex and the $j$th $k$-simplex. Even though in the introduction we mentioned about orientations of higher-order simplices as “Higher-order simplices are usually oriented, ensuring a consistent node arrangement within each simplex, facilitating tasks like determining information flow directions along the edges.”, we did not formally define the orientations of features on the simplicial complex in relation to the reference node ordering. Therefore, we have now specified in the revised manuscript that signs of features on simplices are determined based on the reference node arrangement within each simplex. Additionally, when features are unoriented, we clarify that there is no need for a reference orientation of simplices. In such cases, we consider unoriented incidence matrices.

To this end, we have incorporated the following points in Section 2 (Background) at appropriate locations:

"Each simplex has an orientation defined by a standardized vertex order, typically ascending or descending, establishing a standard node arrangement within each simplex."

"The non-zero entries of an oriented incidence matrix $\mathbf{B}_k$ can be either $+1$ or $-1$, reflecting the relative orientations of the $i$th $(k−1)$-simplex and the $j$th $k$-simplex."

"The sign of feature $\mathbf{X}_k[\sigma_k]$ is determined based on the reference orientation of $\sigma_k$."

"For unoriented features, there is no need for a reference orientation of simplices. In such cases, we consider unoriented incidence matrices. Specifically, the $(i, j)$th entry of an unoriented incidence matrix  $\mathbf{B}_k$ is $1$ if the $i$th $(k−1)$-simplex is a boundary simplex of the $j$th $k$-simplex, and $0$ otherwise."

We believe that these revisions will enhance the comprehension of the orientation equivariance concept within our paper.

Reviewer comment (Theory):
Simplicial awareness, as defined in (Roddenberry et al., 2021), is subtler than the other two properties. Some convolutional-type SNNs in the literature fail to satisfy simplicial awareness if implemented without nonlinear activation functions, due to the fact that the square of the (co)boundary operator is the zero operator. If Theorem 4.2’s assumptions exclude such methods, this should be clarified.

Author response:
We appreciate your feedback and agree that the lack of formal definitions for oriented features and missing to mention the use of unoriented incidence matrices for unoriented features might have caused confusion. We have now clarified these points and expect this to improve the understanding of the theorems and properties related to orientations.

Regarding simplicial awareness, this property requires a model's output to be dependent on all incidence operators. All baseline methods in our paper rely on the powers of lower-adjacent matrices, upper-adjacent matrices, or Hodge Laplacian matrices. As none of these powers approach zero, all baselines inherently exhibit simplicial awareness. However, excluding MPSN, the other baselines do not match the expressivity of the SWL test, as they do not account for feature aggregation from boundary and coboundary simplices, while the SWL test does. Furthermore, since the baselines other than MPSN do not consider aggregation from coboundary simplices, the node embeddings from these models cannot be shown to be more expressive than those from the WL test.

We will supplement Appendix A with the following text for further clarification:
"All baseline methods, including \\texttt{MPSN}, \\texttt{SCNN}, \\texttt{SCoNe}, and \\texttt{S2CNN}, exhibit simplicial awareness as they utilize the powers of lower-adjacent matrices, upper-adjacent matrices, or Hodge Laplacian matrices, none of which approach zero. However, except for \\texttt{MPSN}, these baselines do not match the SWL test in terms of expressivity, as they do not account for feature aggregation from boundary and coboundary simplices, which is considered by the SWL test. Furthermore, since the baselines other than MPSN do not consider aggregation from coboundary simplices, the node embeddings from these models cannot be shown to be more expressive than those from the WL test. We summarize the properties of \\texttt{SaNN} and the existing SNN models as follows:

| Method | Aggregations | More expressive than WL | As expressive as SWL | Simplicial-awareness |
|--------|--------------|------------------------|----------------------|----------------------|
| SaNN   | Precomputed  | Yes                    | Yes                  | Yes                  |
| MPSN   | Sequential   | Yes                    | Yes                  | Yes                  |
| SCoNe  | Sequential   | No                     | No                   | Yes                  |
| SCNN   | Sequential   | No                     | No                   | Yes                  |

Reviewer comment (Theory):
It would be helpful for the authors to briefly survey some of the methods they compare to, and clarify whether Theorems 4.1 and 4.2 apply or do not apply to them.

Author response:
We appreciate your feedback and agree that the lack of formal definitions for oriented features and missing to mention the use of unoriented incidence matrices for unoriented features might have caused confusion. We have now clarified these points and expect this to improve the understanding of the theorems and properties related to orientations.

Regarding simplicial awareness, this property requires a model's output to be dependent on all incidence operators. All baseline methods in our paper rely on the powers of lower-adjacent matrices, upper-adjacent matrices, or Hodge Laplacian matrices. As none of these powers approach zero, all baselines inherently exhibit simplicial awareness. However, excluding MPSN, the other baselines do not match the expressivity of the SWL test, as they do not account for feature aggregation from boundary and coboundary simplices, while the SWL test does. Furthermore, since the baselines other than MPSN do not consider aggregation from coboundary simplices, the node embeddings from these models cannot be shown to be more expressive than those from the WL test.

We will supplement Appendix A with the following text for further clarification:
"All baseline methods, including \\texttt{MPSN}, \\texttt{SCNN}, \\texttt{SCoNe}, and \\texttt{S2CNN}, exhibit simplicial awareness as they utilize the powers of lower-adjacent matrices, upper-adjacent matrices, or Hodge Laplacian matrices, none of which approach zero. However, except for \\texttt{MPSN}, these baselines do not match the SWL test in terms of expressivity, as they do not account for feature aggregation from boundary and coboundary simplices, which is considered by the SWL test. Furthermore, since the baselines other than MPSN do not consider aggregation from coboundary simplices, the node embeddings from these models cannot be shown to be more expressive than those from the WL test. We summarize the properties of \\texttt{SaNN} and the existing SNN models as follows:

| Method | Aggregations | More expressive than WL | As expressive as SWL | Simplicial-awareness |
|--------|--------------|------------------------|----------------------|----------------------|
| SaNN   | Precomputed  | Yes                    | Yes                  | Yes                  |
| MPSN   | Sequential   | Yes                    | Yes                  | Yes                  |
| SCoNe  | Sequential   | No                     | No                   | Yes                  |
| SCNN   | Sequential   | No                     | No                   | Yes                  |

Reviewer comment (Evaluation):
Runtime and asymptotic comparisons in this work are done by excluding the precomputation of features. I think this is misleading, since in practice, the precomputation is certainly part of the computation, especially during inference. Thus, the presented gains seem to be only valid during training, when the features need to be computed only once for many iterations of training.

Author response:
We appreciate your feedback regarding the inclusion of precomputation times in our overall computation assessment. While we agree that precomputation is indeed part of the overall process, it generally constitutes a relatively insignificant portion compared to the cumulative time spent on training and testing. 

It is important to note that our method requires precomputation of features just once before the training begins. On the other hand, conventional SNNs require this computation for every layer, where the number of layers correspond to the neighborhood depth from which features are aggregated, and for each epoch and across all orders of simplices. In Appendix B, we do provide a theoretical comparison of overall computational complexities of SaNN and the existing SNN models. The overall time complexity of the existing SNN models is about $\mathcal{O}\left(T\left(\left(2 N_k^2 D_k+N_k N_{k-1} D_{k-1}+N_k N_{k+1} D_{k+1}\right)+\left(3 N_k D_k^2+N_k D_{k-1}^2+N_k D_{k+1}^2\right)\right)\right)$, which is dominated by the first term containing $2 N_k^2 D_k, N_k N_{k-1} D_{k-1}$, and $N_k N_{k+1} D_{k+1}$, corresponding to feature aggregation in every layer. Since feature aggregation happens only once before training as a pecomputation step, an SaNN model capturing information from $0, \ldots, T$-hop neighborhood of $k$-simplices has a time complexity of $\mathcal{O}\left(T\left(3 N_k D_k^2+N_k D_{k-1}^2+N_k D_{k+1}^2\right)\right)$, which is negligibly small when compared to the computational complexity of the existing SNN models. 

We acknowledge that the time savings during inference may not be as pronounced as those during training, but we assert that the substantial enhancements during the training phase are quite significant, given that training time is the primary contributor to overall computation time, which is the sum of precomputation time, training time and inference time. 
Nonetheless, in response to your suggestion, we have incorporated precomputation times into our analysis   to provide a more comprehensive view of our method's performance. We present here the precomputation and per epoch training time values of SaNN and the per epoch training time values of baseline SNN models for the three tasks at hand. The first terms in the runtime values of SaNN correspond to the precomputation times and the second terms to the per epoch training times.

Trajectory prediction:
| Dataset | Ocean | Synthetic | Planar | Mesh |
|---------|-------|-----------|--------|------|
| ScoNe   | 0.4   | 3.2       | 30.2   | 18.9 |
| SCNN    | 1.9   | 4.2       | 36.3   | 29.8 |
| SaNN    | 0.01,0.1 | 0.06,2.5 | 1.22,26.8 | 0.39,8.1 |


Simplicial closure prediction:
| Dataset      | Enron   | High-school | Primary-school | NDC-classes | Math-sx     |
|--------------|---------|-------------|----------------|-------------|-------------|
| MPSN         | 255     | 413         | 3499           | -           | -           |
| SCNN         | 17      | 401         | 1891           | -           | -           |
| SaNN         | 0.01, 3 | 0.05, 112   | 0.76, 916      | 0.26, 13    | 95.91, 52883|

Graph classification:
| Dataset  | Proteins | NCI1 | IMDB-B | Reddit-B | Reddit-M |
|----------|----------|------|--------|----------|----------|
| MPSN     | 33       | 292  | 46     | 242      | 1119     |
| SaNN     | 3.6, 0.4 | 6, 58| 4, 8   | 6, 45    | 34, 104  |


The values provided represent the training time per epoch. Out-of-memory results are indicated by −. The total training time for SaNN is calculated as follows:
Total training time = Precomputation time + (Number of epochs * Training time per epoch). This total is substantially less than the precomputation times. Furthermore, it is also worth noting that for any neural network model, the larger computational burden lies in the training phase rather than the inference phase. In our study, we have demonstrated both theoretically and experimentally that our training times are significantly reduced compared to other models. Therefore, the overall computational efficiency of our method remains superior.

Reviewer comment (Experiments):
The method requires many hyper parameter choices like hops, T, k, which seem to have different optimal settings on different datasets. The result quality differs substantially depending on the configuration.

Author response:
We acknowledge your concern about the numerous hyperparameters in our method, such as depth $(T)$, order of simplicial complex $(K)$, and hops. It is true that optimal settings can vary across different datasets, potentially impacting the quality of results.

However, it is important to note that hyperparameters like depth $T$ and order $K$ are not unique to our model, but are intrinsic to any model dealing with simplicial complexes. To understand the impact of information from different depths and simplices of varying orders, we conducted an ablation study. Our findings from this study are detailed in Appendix J. We have also summarized key insights from this analysis in Section 5 under "Additional Insights." We believe these observations can guide the selection of hyperparameters for different applications.

Reviewer comment (Evaluation):
I wonder how the method compares to previous methods in inference runtime, when feature precomputation needs to be included.

Author response:
We appreciate your feedback regarding the inclusion of precomputation times in our overall computation assessment. While we agree that precomputation is indeed part of the overall process, it generally constitutes a relatively insignificant portion compared to the cumulative time spent on training and testing. 

It is important to note that our method requires precomputation of features just once before the training begins. On the other hand, conventional SNNs require this computation for every layer, where the number of layers correspond to the neighborhood depth from which features are aggregated, and for each epoch and across all orders of simplices. In Appendix B, we do provide a theoretical comparison of overall computational complexities of SaNN and the existing SNN models. The overall time complexity of the existing SNN models is about $\mathcal{O}\left(T\left(\left(2 N_k^2 D_k+N_k N_{k-1} D_{k-1}+N_k N_{k+1} D_{k+1}\right)+\left(3 N_k D_k^2+N_k D_{k-1}^2+N_k D_{k+1}^2\right)\right)\right)$, which is dominated by the first term containing $2 N_k^2 D_k, N_k N_{k-1} D_{k-1}$, and $N_k N_{k+1} D_{k+1}$, corresponding to feature aggregation in every layer. Since feature aggregation happens only once before training as a pecomputation step, an SaNN model capturing information from $0, \ldots, T$-hop neighborhood of $k$-simplices has a time complexity of $\mathcal{O}\left(T\left(3 N_k D_k^2+N_k D_{k-1}^2+N_k D_{k+1}^2\right)\right)$, which is negligibly small when compared to the computational complexity of the existing SNN models. 

We acknowledge that the time savings during inference may not be as pronounced as those during training, but we assert that the substantial enhancements during the training phase are quite significant, given that training time is the primary contributor to overall computation time, which is the sum of precomputation time, training time and inference time. 
Nonetheless, in response to your suggestion, we have incorporated precomputation times into our analysis   to provide a more comprehensive view of our method's performance. We present here the precomputation and per epoch training time values of SaNN and the per epoch training time values of baseline SNN models for the three tasks at hand. The first terms in the runtime values of SaNN correspond to the precomputation times and the second terms to the per epoch training times.

Trajectory prediction:
| Dataset | Ocean | Synthetic | Planar | Mesh |
|---------|-------|-----------|--------|------|
| ScoNe   | 0.4   | 3.2       | 30.2   | 18.9 |
| SCNN    | 1.9   | 4.2       | 36.3   | 29.8 |
| SaNN    | 0.01,0.1 | 0.06,2.5 | 1.22,26.8 | 0.39,8.1 |


Simplicial closure prediction:
| Dataset      | Enron   | High-school | Primary-school | NDC-classes | Math-sx     |
|--------------|---------|-------------|----------------|-------------|-------------|
| MPSN         | 255     | 413         | 3499           | -           | -           |
| SCNN         | 17      | 401         | 1891           | -           | -           |
| SaNN         | 0.01, 3 | 0.05, 112   | 0.76, 916      | 0.26, 13    | 95.91, 52883|

Graph classification:
| Dataset  | Proteins | NCI1 | IMDB-B | Reddit-B | Reddit-M |
|----------|----------|------|--------|----------|----------|
| MPSN     | 33       | 292  | 46     | 242      | 1119     |
| SaNN     | 3.6, 0.4 | 6, 58| 4, 8   | 6, 45    | 34, 104  |


The values provided represent the training time per epoch. Out-of-memory results are indicated by −. The total training time for SaNN is calculated as follows:
Total training time = Precomputation time + (Number of epochs * Training time per epoch). This total is substantially less than the precomputation times. Furthermore, it is also worth noting that for any neural network model, the larger computational burden lies in the training phase rather than the inference phase. In our study, we have demonstrated both theoretically and experimentally that our training times are significantly reduced compared to other models. Therefore, the overall computational efficiency of our method remains superior.

Reviewer comment (Evaluation):
There do not appear to be any results describing the pre-computation time which should be included in any run-time comparisons which I imagine should scale near-exponentially with graph size and order of simplices considered.

Author response:
We appreciate your feedback regarding the inclusion of precomputation times in our overall computation assessment. While we agree that precomputation is indeed part of the overall process, it generally constitutes a relatively insignificant portion compared to the cumulative time spent on training and testing. 
Nonetheless, in response to your suggestion, we incorporated precomputation times into our analysis to provide a more comprehensive view of our method's performance. We present here the precomputation and per epoch training time values of SaNN and the per epoch training time values of baseline SNN modesl for the three tasks at hand. The first terms in the runtime values of SaNN correspond to the precomputation times and the second terms to the per epoch training times.
Trajectory prediction:
| Dataset | Ocean | Synthetic | Planar | Mesh |
|---------|-------|-----------|--------|------|
| ScoNe   | 0.4   | 3.2       | 30.2   | 18.9 |
| SCNN    | 1.9   | 4.2       | 36.3   | 29.8 |
| SaNN    | 0.01,0.1 | 0.06,2.5 | 1.22,26.8 | 0.39,8.1 |

Simplicial closure prediction:
| Dataset      | Enron   | High-school | Primary-school | NDC-classes | Math-sx     |
|--------------|---------|-------------|----------------|-------------|-------------|
| MPSN         | 255     | 413         | 3499           | -           | -           |
| SCNN         | 17      | 401         | 1891           | -           | -           |
| SaNN         | 0.01, 3 | 0.05, 112   | 0.76, 916      | 0.26, 13    | 95.91, 52883|

Graph classification:
| Dataset  | Proteins | NCI1 | IMDB-B | Reddit-B | Reddit-M |
|----------|----------|------|--------|----------|----------|
| MPSN     | 33       | 292  | 46     | 242      | 1119     |
| SaNN     | 3.6, 0.4 | 6, 58| 4, 8   | 6, 45    | 34, 104  |

The values provided represent the training time per epoch. Out-of-memory results are indicated by −. The total training time for SaNN is calculated as follows:
Total training time = Precomputation time + (Number of epochs * Training time per epoch). This total is substantially less than the precomputation times.

Reviewer comment (Evaluation):
SaNNs are often not outright the winner in terms of prediction accuracies for the tasks displayed in Tables 1 and 3. For example, in Table 1, the SaNN is outcompeted by Projection and Scone on 3/4 of the datasets and the run-time savings of SaNN are not significant enough to justify usage of the SaNN. In Table 3, SaNN is not the leader in 4/5 of the datasets and it is not even the fastest. On the other hand, the time savings against MPSN are quite significant, but since many practitioners of graph learning expect training to take significant amounts of time, accuracy is the topmost priority, so there wouldn't be a strong enough justification to go with a SaNN.

Author response:
We agree that SaNN does not consistently outperform other models in terms of prediction accuracy for certain datasets. However, the true strength of SaNN lies in its substantial runtime improvements and theoretical characterization while maintaining competitive accuracy. This computational efficiency is crucial for practical applications.
The reduced training time that SaNN offers enables more frequent model retraining, which is particularly beneficial for real-time applications. It also ensures scalability for larger datasets. As shown in Table 2, traditional SNN models struggle with large datasets, while SaNN excels, especially with datasets in simplicial closure prediction where existing SNNs face memory constraints.

In Table 3, although SaNN is not the leader in average accuracies for most datasets, the high standard deviations suggest significant overlap in accuracy results, rendering the differences between the top-performing models statistically insignificant. While SPIN is the fastest for graph classification, it is important to note that it is not applicable to applications involving simplicial complexes such as simplicial closure prediction and trajectory prediction.

While accuracy is indeed a priority in many applications, the balance between accuracy and computational efficiency is an equally critical factor in real-world scenarios. SaNN offers a valuable trade-off in this context, making it a viable option, particularly for large-scale applications.

Reviewer comment (Theory):
Is assuming the learnable transformation functions $g_k^{(t)}\cdot)$ are injective too strong? Although the MLPs will be injective, appealing to the Universal Approximation Theorem to declare that $g_k$ can be injectively-approximated is probably not practical.

Author response:
We agree that citing the Universal Approximation Theorem to propose that $g_k$ can be injectively approximated may seem like a strong assumption. This assumption is primarily used for theoretical justification, and we understand its potential limitations in practical applications.
However, it is important to note that we have only used multilayer perceptrons (MLPs) as one possible example of the many functions that $g_k$ could be. It is possible to choose any suitable learnable injective function. Our choice of MLPs is simply an example, and it has indeed performed well in our numerical experiments. Therefore, while the injectivity assumption offers theoretical support, it does not diminish the practical effectiveness of our proposed method.

Reviewer comment (Evaluation):
I may have missed this but are pre-computation times explicitly indicated in the results?

Author response:
We appreciate your feedback regarding the inclusion of precomputation times in our overall computation assessment. While we agree that precomputation is indeed part of the overall process, it generally constitutes a relatively insignificant portion compared to the cumulative time spent on training and testing. 
Nonetheless, in response to your suggestion, we incorporated precomputation times into our analysis to provide a more comprehensive view of our method's performance. We present here the precomputation and per epoch training time values of SaNN and the per epoch training time values of baseline SNN modesl for the three tasks at hand. The first terms in the runtime values of SaNN correspond to the precomputation times and the second terms to the per epoch training times.
Trajectory prediction:
| Dataset | Ocean | Synthetic | Planar | Mesh |
|---------|-------|-----------|--------|------|
| ScoNe   | 0.4   | 3.2       | 30.2   | 18.9 |
| SCNN    | 1.9   | 4.2       | 36.3   | 29.8 |
| SaNN    | 0.01,0.1 | 0.06,2.5 | 1.22,26.8 | 0.39,8.1 |

Simplicial closure prediction:
| Dataset      | Enron   | High-school | Primary-school | NDC-classes | Math-sx     |
|--------------|---------|-------------|----------------|-------------|-------------|
| MPSN         | 255     | 413         | 3499           | -           | -           |
| SCNN         | 17      | 401         | 1891           | -           | -           |
| SaNN         | 0.01, 3 | 0.05, 112   | 0.76, 916      | 0.26, 13    | 95.91, 52883|

Graph classification:
| Dataset  | Proteins | NCI1 | IMDB-B | Reddit-B | Reddit-M |
|----------|----------|------|--------|----------|----------|
| MPSN     | 33       | 292  | 46     | 242      | 1119     |
| SaNN     | 3.6, 0.4 | 6, 58| 4, 8   | 6, 45    | 34, 104  |

The values provided represent the training time per epoch. Out-of-memory results are indicated by −. The total training time for SaNN is calculated as follows:
Total training time = Precomputation time + (Number of epochs * Training time per epoch). This total is substantially less than the precomputation times.

Reviewer comment (Reproducibility):
Although the main principles are clear, I am still confused about the actual architecture/predictive models. In the end we have equation 8, but it describes a representation for each of the $N$ sets of $k$-simplices, each consisting of the $N_k$ simplices. It is unclear how to distill a global prediction out of all these representations, as would be needed for e.g. the classification tasks. Details on how the architectural design for each of the benchmarks is missing.

Author response:
The predictive models, which are used to learn task-specific embeddings from the embeddings of all simplices (as calculated using Equation 8), are detailed in Appendix H.

To use the edge embeddings learnt using SaNN to perform trajectory prediction, we map the embeddings from the vector space of edge embeddings to the vector space of node embeddings by multiplying them by the incidence matrix $\mathbf{B}_1^T$. We consider the node embeddings of only those nodes that have a possibility of being the next in the trajectory, which are the neighboring nodes of the last node in a trajectory. For a trajectory t, which ends at node i, we use a masking function to extract the embeddings corresponding to the neighboring nodes of the node i. We do this for node embeddings from different depths and concatenate them. We then give the combined node embeddings as input to a decoder MLP which maps the node embeddings to a vector space of dimension same as the maximum number of neighboring nodes of the last node in any trajectory in the dataset. The last layer of the decoder is a softmax activation function, the output of which for a trajectory is a vector containing the probabilities with which the neighboring nodes of the trajectory’s last node become the successor node of that trajectory. For all the datasets, we train SaNN in a supervised manner using the cross-entropy loss function. 

To perform simplicial closure prediction for triangles, we concatenate the embeddings of the constituent nodes and edges in the open triangles in the training set. The concatenated embeddings are given as input to a decoder MLP with a sigmoid activation function in the last layer, the output of which is the probability of the open triangles forming a simplex. For all the datasets, we train SaNN in a supervised manner using the binary cross-entropy loss function. Since the datasets are heavily skewed with many negative examples (open triangles that never get closed) than positive examples (open triangles that get closed), we perform oversampling of the positive examples in each training batch and use a weight for positive examples in the binary cross-entropy loss function.
 
To perform graph classification using the embeddings learnt using SaNN, we combine embeddings of the constituent simplices of all orders in the clique-lifted graphs. Since the number of k-simplices in a training example (a clique-lifted graph) is large concatenating embeddings of all the k-simplices in a training example will result in very high-dimensional embeddings. Therefore, we use summation as the readout function at $k$-simplices level. We, finally, use concatenation as the global-level readout function. Specifically, we take a summation of all the node embeddings, edge embeddings, and triangle embeddings in a graph and concatenate the nodelevel, edge-level, and triangle-level embeddings of the simplicial complex to get one global representation for the simplicial complex. The global embeddings of graphs are given as input to a decoder MLP with either a sigmoid activation function or a softmax activation function as the final layer depending on whether the dataset has two or more classes. For all the datasets, we train SaNN in a supervised manner using the cross-entropy loss function, which takes the output of the decoder and the true graph labels as the input.

The hyperparameters used in SaNN to implement the different tasks are summarized in Tables 5, 7, and 9. We have maintained consistency in our experiments by using the same hyperparameters, such as depth ($T$), order ($K$), and hidden dimensions, across all baseline SNN models. This was done to ensure a fair computational comparison. The predictive models for the three tasks are also the same for all the neural network-based models. Due to space constraints, we provided all these details in the Appendix.

To provide further clarity about the hyperparameters used for benchmarks, we have added the following line to Appendix H: 
"For a fair computational comparison, the same predictive models, and hyperparameters, namely depth ($T$), order ($K$), and hidden dimensions, were utilized for all the baseline SNN models.

Reviewer comment (Experiments):
On several occasions the notion of "non-deep baselines" is used. What is meant by this. Could you clarify what non-deep means here, which methods are these?

Author response:
In our paper, the term "non-deep" refers to models that do not utilize deep learning methodologies. Examples include techniques like Projection for trajectory prediction and Logistic Regression for simplicial closure prediction.

The projection based method is used as a baseline for trajectory prediction in Roddenberry et al. (2021). In the projection-based method for trajectory prediction, a mapping that projects the input edge signal corresponding to a trajectory onto the kernel of the Hodge Laplacian is considered. The output is passed through a softmax layer to pick a successor node in the trajectory. 

For simplicial closure prediction, we use logistic regression as a non-deep baseline, following the approach proposed by Benson et al. (2018). In this context, logistic regression takes as its input the harmonic, geometric, and arithmetic means of the three edge weights in the open triangle. This is a method suggested by Benson et al. (2018) in their work.

Reviewer comment (Writing):
In section 2 when presenting the symbols it is mentioned that $k=1,2,\dots,N+1$. Does $k$ always run up all the way to $N+1$?

Author response:
Thank you for bringing this to our attention. You are correct; the variable $k$ does not necessarily run up all the way to $N+1$. This was an oversight on our part. In the revised manuscript, we introduced the notation of the order of the simplicial complex as $K$ and corrected the upper limit of $k$ wherever applicable.

Reviewer comment (Writing):
Just above property 1 it is mentioned "other commonly used sum, mean, or max read-out functions are not injective" I am not fully sure I understand it correctly. The paragraph above explains that sum aggregation is the best injective aggregator, in contrast to mean aggregation. I think the statement that I just quoted is about aggregating over the different $\mathbf{Y}$'s? Perhaps this can be clarified.

Author response:
When we discuss the injectivity of an aggregation function, we are referring to its ability to distinguish between non-isomorphic simplicial complex structures with identical initial features. In this context, the sum aggregation function is effective at producing distinct outputs for distinct structures after aggregation.

The sentence above Property 1, however, is discussing the read-out function which combines features of simplices aggregated from different types of neighborhoods. Here, the sum function may not be injective, as combining different sets of embeddings through summation could result in identical outputs.

Reviewer comment (Presentation):
In the tables: since colors red and blue are used you might as well color the text in the caption as well. I.e. "The {\color{red}first} and {\color{blue}second} best performances ..."

Author response:
Thank you for your suggestion regarding the color coding in the captions of tables. We have incorporated this change in our revised manuscript.

Reviewer comment (Reproducibility):
The insights section says "The deep models are observed to perform exceptionally better than logistic regression", where do I see this? Logistic regression taking what as input? Could this be clarified.

Author response:
In numerical experiments in the paper, we use logistic regression as a non-deep baseline for simplicial closure prediction, following the approach proposed by Benson et al. (2018). In this context, logistic regression takes as its input the harmonic, geometric, and arithmetic means of the three edge weights in the open triangle. This is a method suggested by Benson et al. (2018) in their work.