Paper: Unsupervised Learning via Network-Aware Embeddings (ICLR-2024)

Reviewer comment (Novelty):
The technical depth of the proposed approach is not so deep.

Author response:
We do agree that the technical contribution is shallow. In our intention, the technical contribution is limited to show that a dedicated approach to solving this innovative problem can outperform adapting other embedding methods. In a sense, the simplicity is a feature: it makes the approach easier to understand and shows that there are large gains to be made once we establish that this is a path worth following. The paper aims to establish that this is a path worth following, optimizing the performance is worthwhile to do only once we have established this.

Reviewer comment (Reproducibility):
Simple model to be easily implemented.

Author response:
We do agree that the technical contribution is shallow. In our intention, the technical contribution is limited to show that a dedicated approach to solving this innovative problem can outperform adapting other embedding methods. In a sense, the simplicity is a feature: it makes the approach easier to understand and shows that there are large gains to be made once we establish that this is a path worth following. The paper aims to establish that this is a path worth following, optimizing the performance is worthwhile to do only once we have established this.

Reviewer comment (Theory):
Regarding technical depth, the proposed approach is a combination of the previous approach and is technically somewhat shallow; this paper should show theoretical analyses of the proposed approach.

Author response:
We do agree that the technical contribution is shallow. In our intention, the technical contribution is limited to show that a dedicated approach to solving this innovative problem can outperform adapting other embedding methods. In a sense, the simplicity is a feature: it makes the approach easier to understand and shows that there are large gains to be made once we establish that this is a path worth following. The paper aims to establish that this is a path worth following, optimizing the performance is worthwhile to do only once we have established this.

Reviewer comment (Theory):
The computational costs of t-SNE and DBSCAN are quadratic to the number of data points; what is the theoretical computational cost of the proposed approach?

Author response:
Q: What is the theoretical computational cost of the proposed approach?  

A: The theoretical computational cost of the proposed approach is |O|^2 + |V|^x, with x dependent on the exact Laplacian solver used, but normally x ~ 1.3, decisively below 2. The |O|^2 term comes from the correct observation from the reviewer: tSNE and DBSCAN are quadratic and they are applied to the node attributes O. In most real world scenarios, |O| <<< |V|, we normally have much fewer node attributes than nodes. If that's the case, |O|^2 disappears and the running time is dominated by calculating the GE distances, whose complexity is below |V|^2, as the figure shows.

Reviewer comment (Evaluation):
From the results shown in Section 3.4, the running time of the proposed approach is less than |V|^2—please explain why this happens.

Author response:
Q: What is the theoretical computational cost of the proposed approach?  

A: The theoretical computational cost of the proposed approach is |O|^2 + |V|^x, with x dependent on the exact Laplacian solver used, but normally x ~ 1.3, decisively below 2. The |O|^2 term comes from the correct observation from the reviewer: tSNE and DBSCAN are quadratic and they are applied to the node attributes O. In most real world scenarios, |O| <<< |V|, we normally have much fewer node attributes than nodes. If that's the case, |O|^2 disappears and the running time is dominated by calculating the GE distances, whose complexity is below |V|^2, as the figure shows.

Reviewer comment (Experiments):
Although several GNN-based clustering approaches are listed in Section 1, none of them are compared in the experiment; is the proposed approach more effective than the other approaches?

Author response:
Q: Is the proposed approach more effective than the other [GNN] approaches?  

A: We fully expect it to be so. That is because the referred GNN approaches mentioned in our discussion all share the same issue: they were not defined to originally solve the problem we propose. We are arguing that the adaptation of any of those methods will be outperformed by applying GE, since it is designed to tackle this problem explicitly. We have done some experiments resulting in lower performance with other approaches such as Graph Convolutional Networks and Graph Attention Networks. We have not included them in the paper not to have too many comparisons, which would make the figures less clear. We can consider replacing node2vec with one of these methods, or to expand the appendix with these results.

Reviewer comment (Experiments):
The experiments use only two real-world datasets (the Trade Atlas and the Little Sis datasets); is the proposed approach effective in other real-world datasets?

Author response:
Q: Is the proposed approach effective in other real-world datasets?  

A: There is also a third dataset, Tivoli, which shows the same patterns as the previous two real world dataset. So the answer to this question is yes. We can make an effort to find more datasets, if this is considered crucial.

Reviewer comment (Reproducibility):
Even though the proposed approach is a combination of existing approaches such as autoencoder, t-SNE, and DBSCAN, their parameter settings are not shown in the paper; please describe the detailed parameter settings.

Author response:
Q: Please describe the detailed parameter settings.  

A: We can definitely expand on this on the paper. We overlooked such detail, because we largely use default values. The exception being setting min_samples for DBSCAN to 2, and doing a grid search to find the best performing eps parameter value. We also discuss the setting of the two key parameters for node2vec in Sec 3.1. In light of our answer to reviewer x6pK, we will also discuss tSNE's parameter perplexity better, since we will make a study of it.

Reviewer comment (Experiments):
The clustering with DBSCAN is performed in two dimensional space. If clustering in such a low dimensional space works, then the data may be too simple and too easy to solve. In fact, the true number of categories, which is the target number of clusters, in LittleSis data is small -- just two: either Republican or Democrat. The true number of categories in Trade Atlas and Tivoli are not reported but may be small as well. The findings here might not generalize well to other more complicated datasets.

Author response:
About the comment on 2D spaces to be easy to cluster: that is certainly true. However, ending up with an easy 2D clustering problem is exactly the power of the pipeline. Originally, O -- the data we want to cluster -- has n dimensions, with n being the number of nodes in the graph G. n can be very large if we are looking at a social graph -- up to millions or tens of millions of nodes. With our pipeline, we can make sense of these millions of dimensions, reducing them to a low and manageable number. To us, that is a strength of the method.

About the number of categories of Trade Atlas and Tivoli, we can add this information to the paper. For Trade Atlas, the "product categories" we refer to is the first digit of the hierarchical SITC codes -- there are ten product categories in total. For Tivoli, there are seven categories.

Reviewer comment (Experiments):
The tSNE has a tunable parameter called perplexity. I would like to know whether the results are robust with different perplexity parameter values.

Author response:
We can definitely check the impact of perplexity on the results. We will update the review once we manage to run such test, which we can include in the appendix.

Reviewer comment (Experiments):
It is unclear tSNE is the best choice. The use of tSNE into low, 2 or 3, dimensions may be good for visualization but not for clustering. For clustering, the use of UMAP into relatively higher dimensions can be a better choice.

Author response:
We agree UMAP could be a better choice than tSNE. We can use UMAP without issues -- we already have the code to do so. If the reviewer feels strongly about it, we'll do so. However, we point at the last paragraph of Sec 2.4.3 as to why we didn't do it in the first place: we aren't at this point trying to maximize every piece of our framework. Our focus is on defining this innovative problem definition and showing that a dedicated way to tackle it -- using GE -- outperforms adapting other embedding methods that were not developed with this problem in mind. Plugging in UMAP rather than tSNE will likely produce benefits across the board for all methods, whether they use GE or not, which is not a current focus of the paper.

Reviewer comment (Reproducibility):
It is very unclear how the graph auto-encoder (GAE) is applied in the pipeline. Specifically, for Q1: The authors describe the GAE baseline as follows: "GAE: clusters $O$, after passing it through our graph autoencoder, described in Section 2.4". It is not at all clear how one can "cluster $O$" from the output of the GAE. To cluster $O$, you need to cluster the columns of some matrix with column dimension $d(O)$. Since the output of a GAE is either $n\times d_a$ or $n\times n$, it is not at all clear how one can "cluster $O$" after "passing it through" a GAE. Can the authors elaborate?

Author response:
Saying that "GAE clusters" something is sloppy on our part and we'll fix it. What we mean here is that we build a pipeline which applies GAE to O, obtaining a transformed GAE(O), and then uses DBSCAN on GAE(O) to find the clusters. GAE is applied to each o_x in O. Each o_x is a vector of length n, which get first encoded and then decoded by GAE to obtain a new GAE(o_x), again of length n. GAE(O) is the collection of such vectors, which is a matrix of dimension n x d(O), as requested -- because we have d(O) o_x (and GAE(o_x)) vectors.

Reviewer comment (Reproducibility):
Q2: In S2.4.2, the authors write "We could apply tSNE directly to the GAE output." This lacks a formal description, and it is ambiguous how this could actually be achieved. We would need to apply tSNE to some $k \times d(O)$ matrix, which would result in a tSNE-estimated $2 \times d(O)$ matrix. How does the GAE output allow for this?

Author response:
Hopefully, the answer above helps with this question as well. Since GAE outputs GAE(O), an n x d(O) matrix, then this can be easily fed into tSNE to reduce its dimensions to 2 x d(O).

Reviewer comment (Reproducibility):
Q3: Figure 2 implies that somehow the GAE output is used in the distance metric computation. However, this is never described (even vaguely). The formula for the distance metric (given in Section 2.3) includes only the original attribute data $O$ and the inverse Laplacian $L^{+}$, which is derived from the original graph $G$. How does the GAE output feed into this?

Author response:
This depends on the actual pipeline used. Fig2 shows the full pipeline. Here we first apply GAE to transform O into GAE(O). Then, to estimate the distance between two GAE(o_1) GAE(o_2) vectors we use the formula the reviewer mentioned. Since O and GAE(O) have the exact same shape, they can be used interchangeably. In Sec 2.3 we choose to use O not to have to introduce too many details that might get in the way of understanding what the measure is doing. We can of course change this if the reviewer finds it appropriate.

Reviewer comment (Reproducibility):
Q4: The authors write "We could apply tSNE directly to the GAE output. However that would mean that tSNE is seeking for the best representation of the data in an Euclidean space, which is not appropriate because we know the dimensions of our observations are related to each other in $G$". The plausibility of this reasoning is hard to evaluate because we don't know how the GAE is being applied. Depending on how the GAE fed into the tSNE module, tSNE might not actually be applied in a "Euclidean" space, because the output of the GAE directly convolves the node features with the graph structure. Can the authors clarify this statement and the application of GAE itself?

Author response:
Hopefully, our A1 helps with this question as well. We also admit that it is true that GAE convolves O and G. So we should point this out in the text. What we still argue is that applying our GE function to GAE's output is, on average, beneficial, as Table A1 shows (compare GAE+GE+tSNE with GAE+tSNE rows).

Reviewer comment (Experiments):
Q6: While the approximation to $L^{+}$ seems efficient (see Section 3.4), the authors did not explore how the approximation affects the performance of the algorithm. Could the authors replicate the simulations and real data analyses using the approximation to $L^{+}$ rather than the exact solution? This would be a welcome addition to the appendix.

Author response:
We can certainly do that. There is also a recently-appeared note on arXiV (https://arxiv.org/abs/2310.11222) showing that the approximations are in the order of 10^-9 (Sec 5.1, Tab1), which we consider negligible.

Reviewer comment (Evaluation):
Q7: From a methodological perspective, the conclusions that a practitioner can draw from the empirical study are very limited. This is because the baselines of the proposed pipeline do almost as well as the pipeline on the real-world data. For each dataset, two out of the top-3 scoring approaches do not use the proposed graph distance at all. On each dataset, ablations that do use the graph distance have at most middle ranks, and one of the scores for the full pipeline (GAE+GE+tSNE) is near-random performance (see Fig 4(a)). Can the authors further justify the use of their pipeline in light of these results?

Author response:
Our justification is that the application of GE, our distance function, is always beneficial. The top scoring method for all real world datasets always uses GE. What makes the results a bit more confusing is that GAE is not always beneficial and could sometimes be harmful. We think this is due to the amount of noise in the data (as Fig3b shows: adding GAE to GE+tSNE -- orange line -- is only beneficial for high values of sigma -- the noise level). So the justification would be to use GAE+GE+tSNE only if the levels of noise are high, and GE+tSNE when they are not. Using this guideline, one would always achieve top performance in the real world cases we consider in the paper.

Reviewer comment (Theory):
The mathematical statement of the problem definition is unclear: "we look for a function f : G × O → P returning the partition P such that arg minO δ = f(G, O), δ being the function calculating the distance between pairs of observations on G." Nothing is clear in this definition. For example, what does it mean to apply a function f on the whole of G and O; where is the pair of observations on which the function δ is applied; what exactly is the stated objective expressed as arg minO δ = f(G, O) – are we looking for a set O or a member of O; and what should the entity we are looking for minimize?

Author response:
We agree with the reviewer that the mathematical formulation of the problem could be done better. Part of the confusion might stem from the fact that the problem definition is tricky, as it is very different from what it is normally done with embeddings on graphs. We would encourage to read reviewer L2DL first strength point, since they understood clearly the innovative aspect of it.

We now address the second set of questions, since we think they are the most important to understand the method, and then we can try to improve the mathematical notation accordingly.

Reviewer comment (Writing):
Even the explanation offered in words is still unclear despite being a bit better than the mathematical statement: "we want to find the partition P of O such that the graph distance δ over G of observations within the same group in P is minimized – excluding trivial solutions that put each observation in a singleton cluster. This definition hinges on δ: the ability to calculate the graph distances between two o1, o2 ∈ O." The following issues remain: what is the definition of a partition; what is the newly named graph distance δ; which exact distance, out of all pairs in a group, would be minimized; and how are trivial solutions to be excluded?

Author response:
Q: What is the definition of a partition?  
A: A partition for us is a grouping of node attributes o in O. Let us suppose we have four node attributes in O: o_1, o_2, o_3, o_4. A valid partition could be one that says that o_1 and o_2 belong to one cluster, and o_3 and o_4 belong to another.

Q: What is the newly named graph distance δ?  
A: In that definition, δ stands as any distance function that takes as input two node attributes (o_1 and o_2) and a graph G. It will return a scalar which represent the distance between o_1 and o_2, over the edges of G. One can think of δ as an Euclidean distance, where the spatial dimensions are related to each other, with G providing these relationships. Movements along dimensions connected in G contribute less to the distance than movements along dimensions not connected in G.

Q: Which exact distance, out of all pairs in a group, would be minimized?  
A: This would be up to the chosen clustering algorithm to decide -- we could have single, average, or complete linkage, or other options. It is a practical question that we feel is not necessary to address in the problem definition itself, because we are describing the problem in general terms (although we are very much open to modify this and to focus on a specific quality function, if the reviewer thinks this is important).

Q: How are trivial solutions to be excluded?  
A: Same as above, we want to introduce the problem in the most general way possible -- akin to define what clustering is, rather than focusing on a specific sub-problem. There could be many ways to exclude trivial solutions. Again, we can modify this part to be more specific, if the reviewer thinks this would strengthen the paper.

Reviewer comment (Novelty):
The biggest problem with this paper is that the model innovation is not enough. In short, the authors propose a distance metric that considers graph structure based on the scheme of Coscia et al. (2020) and apply it to t-SNE to reduce feature dimensions, thereby achieving the goal of clustering node attributes. From my perspective, this does not meet the standard of an academic paper that will be presented at a prestigious conference.

Author response:
Thank you for your comments. From our understanding, most of the weight that leads the reviewer to give a strong rejection is on point 1. We see points 2 to 5 as easily fixable with simple text editing (we can lift up the main objective description, explain better the role of AE, edit the title, and separate the part of the introduction where we discuss related works into its own section).

About point 1, our perspective is that we agree that the technical contribution by itself is not strong. However, the real contribution of the paper, to us, lays into an innovative problem definition (clustering node attributes using the topology of the network to define the relationships between the dimension of analysis) and the technical contribution represents the only approach specifically dedicated to solve this problem.

We would invite to consider reviewer L2DL first strength point, which acknowledge that this is an innovative and worth exploring problem space.

Reviewer comment (Writing):
The expression in this paper is not concise enough. It is not until the fourth paragraph of the introduction that the main objective of this paper is revealed. The main contribution of the method is the generalized Euclidean proposed in Section 2.3, and many other redundant contents need to be deleted.

Author response:
Thank you for your comments. From our understanding, most of the weight that leads the reviewer to give a strong rejection is on point 1. We see points 2 to 5 as easily fixable with simple text editing (we can lift up the main objective description, explain better the role of AE, edit the title, and separate the part of the introduction where we discuss related works into its own section).

Reviewer comment (Writing):
The expression in this paper is not clear enough. For example, why clean noise, and why can AE clean noise? Why don't other models consider this, and what is the motivation for choosing AE? These are not explained clearly.

Author response:
Thank you for your comments. From our understanding, most of the weight that leads the reviewer to give a strong rejection is on point 1. We see points 2 to 5 as easily fixable with simple text editing (we can lift up the main objective description, explain better the role of AE, edit the title, and separate the part of the introduction where we discuss related works into its own section).

Reviewer comment (Presentation):
The authors should have a section introducing related work.

Author response:
Thank you for your comments. From our understanding, most of the weight that leads the reviewer to give a strong rejection is on point 1. We see points 2 to 5 as easily fixable with simple text editing (we can lift up the main objective description, explain better the role of AE, edit the title, and separate the part of the introduction where we discuss related works into its own section).