From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The work presents a way of decomposing a symmetric adjacency matrix using a Sigmoid and two lower rank matrices representing vectors for homophilous and heterophilous communities. Reviewers feel this is an important area with growing interest, although some earlier work Generalized RDPG also exists which the authors subsequently addressed. Reviewers also liked the model, and arboricity is a good concept to use for sparseness.  The theory shows such embeddings exist for matrices of bounded arboricity. Some experiments are done although the comparisons are limited to some earlier techniques. The paper is well written and motivated.

This is interesting work focussing on the theoretical contribution. The theory is motivated by various recent prior contributions, Peysakhovich & Bottou (2021) and Chanpuriya et al. (2020). Community detection and link prediction are demonstrated for five mid-sized data sets. Analysis of results w.r.t. heterophily on real-world data sets is not done, although as the authors say, a clear definition of heterophily is not currently accepted.  Note also, link prediction methods not based on matrix factorisation can be superior and have not been compared.

Support for acceptance at ICLR was not strong. Reviewers had suggested a number of improvements for discussion and experimental work.  My impression is that the focus of the presentation should be reworked since the core is solid: perhaps trying to better define heterophily and doing some studies on the real data sets to show that working.  Also studies of different model dimensions $K_B$ and $k_C$ could be demonstrated since they are never known in advance.  You have some informed comments here in the response.  Setting these using ground truth would not accepted by typical experimentalists.
justification_for_why_not_higher_score: The paper has a good solid core, a nice model and a good theory.  Many issues and questions remain, though in experimental work, elucidating what heterophily is and really highlighting how the algorithm helps the community.
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: The reviewers better explained their position:  lack of modern comparisons, proper work, and the fact that it is an important problem. 
The authors Theorem 4.2 is the main contribution.
The reviewers had addressed a number of issues but yet questions about experimental work remained.
Reviewer who bought up Generalized RDPG work felt they didn't do a good enough job of addressing it in their revised paper.
For link prediction, GNNs seem to be way ahead now of matrix factorisation methods, which makes the interest in these approaches less.

From: ICLR.cc/2023/Conference/Paper3356/Authors
title: Response to Post Reviewer-Discussion Update
comment: Thank you for explaining your reasoning for lowering your score. 

> the similarity to the work on Generalized RDPG was brought up and this area of work is not compared in enough detail considering the similarity with the work on homopholic and heterophilic relations

As we have stated elsewhere, this work is broadly about the benefits and considerations when simultaneously incorporating nonnegativity, heterophily / non-PSDness, and nonlinearity into a graph factorization. Our work lies at the intersection of these three aspects, but there is a lot of prior work that touches on one or two of them. We are currently citing around 40 works in the related work section, so in the interest of space, we discuss them to varying extents, hoping to paint a concise but holistic picture of prior factorization-based models and how they relate to ours. The means of incorporating heterophily in GRDPG is somewhat related to those in Peysakhovich & Bottou (2021), Miller et al. (2009), and, to a lesser extent, Nourbakhsh et al. (2014). In particular, we believe the overlap between our work and that of Peysakhovich & Bottou, which also discusses heterophily quite extensively, largely encompasses the overlap between our work and the GRDPG paper, which is why we discuss the former in more detail. All of these works allow for heterophily by letting the factorization be non-PSD in some way, so we group their discussion together. (More specifically, the $\boldsymbol{B} \boldsymbol{B}^\top - \boldsymbol{C} \boldsymbol{C}^\top$ and $\boldsymbol{V} \boldsymbol{W} \boldsymbol{V}^\top$ sort of factorizations are essentially equivalent to GRDPG's $\boldsymbol{X} \boldsymbol{I}_{p,q} \boldsymbol{X}^\top$ up to some simple algebraic manipulations, at least in the absence of nonnegativity constraints.)\
To be clear, like the authors of the GRDPG paper, we are not claiming objective superiority of our factorization, and the amount of discussion of each prior method reflects a holistic relevance to our focuses rather than some such objective ranking. Specifically, in this work, a major focus is low-rank expressiveness guarantees. For our arboricity-based bound (as well as prior bounds based on max-degree), it is essential to have a nonlinearity, which is not present in GRDPG, hence the briefer discussion of that work. Also, on the empirical side, as we note in the response to Reviewer pv1p, SVD is the closest that any nonlinearity-less low-rank factorization can approximate an adjacency matrix (in terms of Frobenius error), and we find that our model can be fit to be significantly closer than this in experiments on real-world graphs. (On the other hand, the lack of a nonlinearity in GRDPG facilitates interesting convergence guarantees related to spectral theory.) Note also that neither the GRDPG paper nor that of Peysakhovich & Bottou discuss nonnegativity of the factors, hence the focus elsewhere on BigClam, which does discuss nonnegativity. We would add to the final related works section some more discussion of any specifics (discussed above or otherwise) that reviewers believe are important, though we would have to be selective in the main paper due to space constraints.

> There was also some questions about the bound, to be clarified.

> ensuring that the real-data experiments are strongly supporting the claims (datasets are those which have heterophilic and homophilic relations, and the comparable methods are state-of-the-art)

We are happy to answer any questions about the bound. If the questions relate to Reviewer N53t's comments about tightness of the arboricity bound in practice, perhaps some of them have been addressed in that response. Similarly, in that response, we have tried to address broad concerns about empirical evaluation which were raised by the same reviewer, and we are happy to take further questions. As before, we note that we primarily emphasize the conceptual/theoretical contributions of this work rather than empirical ones.

---

Nourbakhsh, Farshad, Samuel Rota Bulo, and Marcello Pelillo. "A matrix factorization approach to graph compression." 22nd International Conference on Pattern Recognition. IEEE, 2014.

Miller, Kurt, Michael Jordan, and Thomas Griffiths. "Nonparametric latent feature models for link prediction." NeurIPS, 2009.

Peysakhovich, Alexander, and Leon Bottou. "An attract-repel decomposition of undirected networks." arXiv preprint arXiv:2106.09671 (2021).

From: ICLR.cc/2023/Conference/Paper3356/Authors
title: Response to Follow-up
comment: Thank you for following up. Yes, we primarily emphasize the conceptual/theoretical contribution of this work as opposed to the empirical. To respond to your question, in some informal experiments on stochastic versions of each of the factorization-based algorithms, we generally found that they all see some reduction in performance in terms of reconstruction error, but still our method generally achieves the closest fit. In other words, we find that the greater theoretical capabilities of our model to fit graphs are also reflected in empirical results with the training algorithms, both the (full-loss / quadratic-time) version and the (sampled-loss / stochastic / scalable) version. In the paper, we focus on the quadratic version since we are more interested in the capacities of the different models, as opposed to the training algorithm (that is, we use the best possible training algorithm to approximate how well the different models possibly could be fit, without regard to scalability), and we only mention the stochastic version to show that a scalable extension is possible.

From: ICLR.cc/2023/Conference/Paper3356/Reviewer_N53t
title: Post-review discussion
comment: Thank you for clarifying and addressing the concerns in my review. I agree that this work is more conceptual than practical especially along the direction of factorization-based embedding methods. Despite this, I have a follow-up question: how does it perform when you use the stochastic version?

From: ICLR.cc/2023/Conference/Paper3356/Authors
title: Response to Follow-up
comment: Thank you very much. The kind words about our work are much appreciated.

From: ICLR.cc/2023/Conference/Paper3356/Authors
title: Reply to Follow-up
comment: Thank you for following up.

> I did not notice any changes to the text after equation 5, maybe accidentally not added?

Apologies, indeed, we created this sentence, but it was not added in the upload above due to a version control error on our part. The sentence was as follows:

As a rough outline of the argument that follows, we will need $6k$ columns in the new factors $\boldsymbol{B},\boldsymbol{C}$, up from the $k$ columns in $\boldsymbol{X},\boldsymbol{Y}$, because accounting for the possible asymmetry in $\boldsymbol{X} \boldsymbol{Y}^\top$ will double the required columns, and accounting for the nonnegativity of $\boldsymbol{B},\boldsymbol{C}$ will triple the required columns.

We will be sure to include it in the final version.

> a mention of setting the kB + kC clusters with parameter tuning would be good, to clarify that this is problem (data) dependent

Thank you for the suggestion. For the experiments in the paper, we we set the total number of clusters to the ground-truth, and we set the $k_B$/$k_C$ split to 50/50. We mention this in the paper, but we will be sure to clarify the data-dependent nature of this selection, and try to include some of the ideas we note in the response for when ground-truth is not available.

From: ICLR.cc/2023/Conference/Paper3356/Reviewer_s4Ez
title: Post Review Discussion
comment: Thank you for elaborating on what would happend if BigClam, with only positive latent features would try to model the heterophily in the example, this is understable.

It has been very insightful to read this work and this contribution is very nice addition to the factorisation models, especially as heterophily is a topic that is discussed often in many graph related papers.

From: ICLR.cc/2023/Conference/Paper3356/Reviewer_s4Ez
title: Post Review Discussion
comment: Thank you for clarifying.

I did not notice any changes to the text after equation 5, maybe accidentally not added?

If not already done a mention of setting the kB + kC clusters with parameter tuning would be good, to clarify that this is problem (data) dependent.

From: ICLR.cc/2023/Conference/Paper3356/Authors
title: Response to Reviewer s4Ez Part 2
comment: **Q5**: Add reference to “Community Detection in Attributed Graphs: An Embedding Approach”

**Response**: Thank you for the suggested reference, which we have integrated into the paper. As you note, this work focuses on a model which assumes homophily, in contrast to ours.

**Q6**: In Figure 3 the learnt features are shown, and the proposed method has captured the ten cities and the additional 2 gender features. Big Clam only learns the ten cities; have you tried with 20 features and BigClam, as this has the potential to learn each combination of GENDER X CITY as an independent feature element?... if the other methods, like BigClam have more features, like 20 in the synthetic example, are they able to learn as well given the extra capacity?

**Response**: This is a great question. First, we note that BigClam indeed fails to represent the synthetic graph even with a 20-dimensional embedding: we run this experiment and plot the result in Figure 2 (right). Regarding your specific idea of what would happen if BigClam learned each combination of Gender x City as a separate feature, say, in a binary feature matrix $\boldsymbol{B} \in \\{0,1\\}^{n \times 20}$: When you calculate $\boldsymbol{B} \boldsymbol{B}^\top$, you will notice that it comprises several blocks along the diagonal, corresponding to people in the same Gender x City group being linked. The heterophilous behavior of gender in this example, which involves the dominance of off-diagonal entries, necessitates a non-positive semidefinite factorization.

That being said, SVD is indeed capable of fitting the graph with 20 features, but not 12 like our model. The presence of the sigmoid nonlinearity in our model makes it combinatorially more powerful than SVD, which has no nonlinearity. In the motivating example, our method requires 2+10=12 dimensions to capture the graph structure, compared to 2\*10=20 for SVD, which indeed is not a large difference. But the difference would be much larger with additional latent variables beyond 2 genders and 10 cities (e.g., 2+10+5+8 vs 2\*10\*5\*8).


It is these limitations that motivate this work, which is broadly about the benefits and considerations when simultaneously incorporating nonnegativity, heterophily (i.e., no PSD assumption), and nonlinearity into a graph factorization.

**Q7**: Adding file versions as text in the README or a requirements.txt can solve this problem, including the Python version.

**Response**: This is a good suggestion – we will add this information in the final version.

**Q8**: I would reconsider the example of only heterogeneous dating sites… How about an example with fraudsters and customers like in this review [Zheng et al. 2022]? Could it even be enough to just say that in a same-gender dating site the example would actually be homophilic between genders so we use a hetersexual example?

**Response**: Thank you for the thoughtful suggestion. We have added the suggested note about how a mostly homosexual dating app would not exhibit heterophily in this sense. If the ethics review suggests further changes, we will change to the fraudsters/customers example from Zheng et al. – thank you for this reference.

---

Peysakhovich, Alexander, and Leon Bottou. "An attract-repel decomposition of undirected networks." arXiv preprint arXiv:2106.09671 (2021).

From: ICLR.cc/2023/Conference/Paper3356/Authors
title: Response to Reviewer s4Ez Part 1
comment: Thank you for the extensive review. We address the questions and concerns piecewise:

**Q1**: Fig. 1 - can you add some details on the sampling?

**Response**: As suggested, we have now included some clarifying text in the caption of Figure 1 to make it more clear and understandable. The “Expected Adjacency” on the left is a matrix of probabilities that edges appear, with entries in $[0,1]$. The “Sampled Adjacency” on the right is a binary matrix, with entries in $\\{0,1\\}$, for an unweighted graph which was sampled based on that matrix. In particular, letting $\tilde{\boldsymbol{A}}$ be the former matrix and $\boldsymbol{A}$ be the latter one, for $i \leq j$, we have $\boldsymbol{A}\_{ij} \sim \textrm{Bernoulli} (\tilde{\boldsymbol{A}}\_{ij}) $. 

**Q2**: Could you elaborate on why kB + kC = 6k is required?

**Response**: Given the logits of a rank-$k$ factorization of a graph in the LPCA model, that is, the $\boldsymbol{X} \boldsymbol{Y}^\top$, we first symmetrize the logits, i.e., we use the mean of $\boldsymbol{X} \boldsymbol{Y}^\top$ and $\boldsymbol{Y} \boldsymbol{X}^\top$. Note that increases the rank of the factorization by a factor of at most $2$, up to $2k$. We then must convert the factorization with unconstrained $\boldsymbol{X}, \boldsymbol{Y}^\top$ into one with nonnegative factors. As shown in Lemma 4.3, this can increase the rank of the factorization by a factor of at most $3$, up to $6k$. We have added some clarifying text outlining the preceding right after Equation 5 where $k_B + k_C = 6k$ is stated.
Also, please note that this is only for the theoretical results with our proposed nonnegative symmetric factorization.

**Q3**: If I understand correctly, homophilic clusters, in B, are found as those with positive eigenvalues and heterophilic clusters, in C, those with negative eigenvalues?

**Response**: Exactly. And this is the salient difference between our work and, e.g., BigClam, which does not include some equivalent of $\boldsymbol{C}$.

**Q4**: How to choose the kB and kC parameter values? Can these eigenvalues above somehow help with this, although for large data this probably not computationally feasible, can you comment?

**Response**: It may be possible to automatically select a $k_B/k_C$ split by using Algorithm 1 (note that this algorithm does not require $k_B,k_C$ as input), then discarding columns of the factorization with least contribution, e.g., to the Frobenius norm of the matrix.
However, we find that it generally does not make much of a difference on real-world graphs relative to a half/half split. Related to your suggestion, this may be reflected in the fact that large real-world networks tend to have a roughly equal number of extremal negative and positive eigenvalues.

If you are referring to selecting the total number of clusters $k_B+k_C$, this is likely best set based on the application (e.g., as a hyperparameter set by cross-validation), though it may be possible to also set within our framework with a heuristic like, e.g., the elbow method, based on the drop in Frobenius error as embedding dimensionality is increased.

From: ICLR.cc/2023/Conference/Paper3356/Authors
title: Response to Reviewer pv1p
comment: Thank you for the suggestions. We have carefully addressed your comments in the updated paper.

**Q1**: Missing references to random dot product models

**Response**: Thanks for pointing out these related papers. We have now included references to these papers in the revised draft, and included another relevant one (Nourbaksh et al. 2014) that we found as well.

**Q2**: Discuss relation of model to Generalized RDPG and other dot product models

**Response**: Thanks for this suggestion. We have added a mention of the Generalized RDPG model and positioned it relative to the others we discuss in the updated draft. The GRDPG is an interesting approach to integrating heterophily into these models. We note that the RDPG models do not have a nonlinear linking function (e.g, $\sigma$, the logistic function as in our model), which is crucial to our representation results. This includes both the theoretical exact representation guarantees, as well as the empirical error minimization results in Figure 4, given that SVD lower bounds Frobenius approximation error for all rank-$k$ factorizations without a nonlinearity, including RDPG and GRDPG. In this regard, the model of Miller et al. (2009) is closer to ours, which we have also discussed in paper. Broadly, this work is about the benefits and considerations when simultaneously incorporating nonnegativity, heterophily (i.e., no PSD assumption), and nonlinearity into a graph factorization, so while a lot of prior work touches on one or two aspects, there is fairly little at the intersection of all three.

---

Nourbakhsh, Farshad, Samuel Rota Bulo, and Marcello Pelillo. "A matrix factorization approach to graph compression." 22nd International Conference on Pattern Recognition. IEEE, 2014.

Miller, Kurt, Michael Jordan, and Thomas Griffiths. "Nonparametric latent feature models for link prediction." NeurIPS, 2009.

From: ICLR.cc/2023/Conference/Paper3356/Authors
title: Response to Reviewer N53t
comment: Thanks for your comments. We address the comments, which fall into two categories:

**Q1**: Applicability to real-world datasets

**Response**: First, we note that this work is primarily conceptual. Broadly, it is about the benefits and considerations when simultaneously incorporating nonnegativity, heterophily (i.e., no PSD assumption), and nonlinearity into a graph factorization. There is a line of work on limitations of factorization-based models for representing graphs, and we mainly intend our work to advance conceptual/theoretical understanding in this area. The empirical section is simply meant to suggest that capturing heterophily can be beneficial for factorizations in downstream tasks. As an aside, this is why we only compare to factorization-based methods for interpretable link prediction. The stochastic version of the training algorithm is noted mainly as a contrast to some other prior algorithms, which do not admit a scalable version - we employ the non-stochastic version for the empirical results in this work. Relatedly, we work with small to medium sized datasets, which may be why the arboricity/degeneracy bound seems insignificant. Degeneracy can be far more tractable for large real-world networks, see, e.g., Table 2 in Noujan et al. (2021). We also reiterate that degeneracy, though it can be computed relatively easily, is only an upper bound on arboricity, and degeneracy is an upper bound on max degree.

**Q2**: Can you quantify heterophily in real-world datasets?

**Response**: Apologies for the confusion. To our knowledge, there is no well-accepted quantitative definition of heterophily that is applicable to these experiments. There is a definition in some prior work (see GeomGCN below) that has seen popularity in related papers: it defines heterophily in terms of ground-truth class/label data (i.e., what fraction of edges are between nodes of the same class), but this is not applicable to our setting of overlapping communities and multiple labels per node. As our illustrative example suggests, it may be possible to naturally define heterophily without reference to ground-truth labels. In fact, we hope that this work can serve as a basis for such definition of heterophily, since there is a significant difference between how well our model and BigClam, the salient distinction between which is the inclusion of a heterophilous term in the factorization, can match a given adjacency matrix (as in our Figure 4).

As you note, aside from the preceding, we do include a fairly concrete and very intuitive synthetic demonstration of modeling heterophily in Figures 1-3, and we discuss heterophilous aspects of some of the real-world datasets in Appendix A.2. Beyond matching the adjacency matrix, incorporating a heterophilous term also seems to improve the ability to predict links and ground truth communities (Figure 5).

---

Pashanasangi, Noujan, and C. Seshadhri. "Faster and generalized temporal triangle counting, via degeneracy ordering." KDD, 2021.

Pei, Hongbin, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. "Geom-GCN: Geometric Graph Convolutional Networks." ICLR, 2019.

From: ICLR.cc/2023/Conference/Paper3356/Reviewer_N53t
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper proposes an exact nonnegative matrix factorization method that provides interpretable node embeddings (interpretation as the cluster/community assignment). The authors prove that under the constraint of embedding dimensionality, the embeddings can reconstruct adjacency matrix exactly. A training algorithm is provided and shown to be effective in community detection and interpretable link prediction.
strength_and_weaknesses: Strengths:
- The proposed model is interpretable to provide node embeddings that can be interpreted as cluster assignments.
- The paper theoretically proves under certain embedding dimensionality, the node embeddings can be used to perfectly reconstruct the input adjacency matrix.
- The graph model can explicitly capture the homophily and heterophily of the graph.
- An illustrative example is provided in the introduction to help understand the motivations of this work.

Weaknesses:
- It's not clear whether the embedding dimension condition on arboricit is tighter than the existing degree-based theorem.
- The proposed algorithm has a high complexity given that it computes the reconstructed dense adjacency matrix in each iteration (BB^T-CC^T).
- Interpretable link prediction is not a new application, but in the experiments only few methods are compared. 
- The heterophily of the datasets is not clearly justified.
clarity,_quality,_novelty_and_reproducibility: The novelty is good since it studies the exactness of the embeddings on graphs with both homophily and heterophily.
The paper overall is clear and not hard to follow.
It's not clear whether the results presented in the paper is implemented based on the stochastic version or non-stochastic version.
summary_of_the_review: To further justify the weaknesses:
- It's not clear whether 4 * \alpha^2 is a smaller number than 2c (where c is the max degree). In appendix, the authors provide the value of degeneracy, but based on that number, 4 * \alpha^2 is definitely much larger than 2c. In this way, I don't think the bound based on arboricit has a wider applicability.  
- Regarding the heterophily of the datasets, though the authors illustrate the heterophily intuitively, it's not quantified. Thus, it's unable to justify whether the proposed method can model heterophily in practice.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3356/Reviewer_pv1p
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this paper, the authors propose a representation model in the framework of describing an adjacency matrix through matrix factorization. They also prove a result regarding exact matrix representation, where the needed size of the embeddings can be described in terms of the arboricity of the graph. This improves previous results in terms of applicability to real graphs.
Finally, the authors propose an alternative learning method for the factorization based on gradient descent, which is then used for downstream tasks in the experimental section.
strength_and_weaknesses: The strength of the paper is the result regarding arboricity and the factorization of the adjacency matrix.

However, I find a significant weakness. There is a quite big community working in graph models via factorization, or "dot product graphs", which is not acknowledged in the paper. I'm sure this is not intentional, but the amount of work of these groups is quite large, and published in journals and conferences very related to this one (meaning that this is not a case of similar work but in a very different community) like NeurIPS or PAMI.
Namely, the RDPG model is widely studied and not mentioned in the paper. The model is almost exactly like the ones described in this work, and moreover, there is a version called Generalized RDPG which takes into account heterophily and homophily, and its formulation is very related to the one proposed in this paper (in equation 3).

I think the Generalized RDPG model is described in this paper:
P. Rubin-Delanchy, J. Cape, M. Tang, and C. E. Priebe, "A statistical interpretation of spectral embedding: The generalized random dot product graph," 2017

and some other papers on the subject are:
 - E. Scheinerman and K. Tucker, "Modeling graphs using dot product representations," Comput. Stat.
 - A. Athreya, D. E. Fishkind, M. Tang, C. E. Priebe, Y. Park, J. T.
Vogelstein, K. Levin, V. Lyzinski, and Y. Qin, "Statistical inference on random dot product graphs: A survey,"
 - I. Gallagher, A. Jones, and P. Rubin-Delanchy, "Spectral embedding
for dynamic networks with stability guarantees," Advances in Neural
Information Processing Systems 34, NeurIPS, 2021.
 - B. Marenco et. al, Online change-point detection for weighted and directed random dot product graphs. IEEE Transactions on Signal and Information Processing over Networks, 2022.

clarity,_quality,_novelty_and_reproducibility: The paper is well written in general, the results are clearly stated (and the proof is easy to follow). As far as I know, this result (regarding arboricity) is original and is an interesting extension of existing work. However, the formulation of the model is very related to existing work, which is not described in the paper (see previous section).

The code is included as supplementary material, so this is a good point too.

summary_of_the_review: The paper proposes interesting ideas, but the (unintentional) lack of references to very similar works makes it very hard to accept it in this first version.
I'm sure the authors will take into account these (and maybe other) related papers and will come up with a new version, which would be a very nice paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3356/Reviewer_s4Ez
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper addresses the issue with representing (esp. heterophilic, power law) graphs by factorising the adjacency matrix into a combination of homophilic and heterophilic factors that are interpretable. Lower bounds based on logisitic principal components analysis (LPCA) is improved removing assumptions on the max degree and replacing with the arboricity of the graph. A new graph model is proposed with symmetric and non-negative factorisations based on the decomposition mentioned and a simple gradient descent alogirthm is proposed to optimise the paramters. The method is compared to a baseline SVD decomposition and recent work using a similar method, BigClam. Results on synthetic and real-world data are reported with promising results.

strength_and_weaknesses: 
This paper addresses the issue of modelling graphs with both homophily and heterophily in a elegent decomposition that is both simple and effective. The new lower bound is novel and seems valid to the best of my knowledge but I could not follow all steps in detail due to time constraints and lack of deep knowledge in the area. The algorithm to optimise the model is simple and results looked promising. This is very relevant work, and a hot topic in graph related domains including GNNs for example.

I do wonder about the comparison with the other methods, as in the comparison with synthetic data, the BigClam method was limited to 12 factors, when I think using 20 factors would be interesting, and may lead to improved fitting of the BiClam method. Although the model is elegant, the selection of the hyper-parameters (how many homophilic and how many heterophilic factors) is left open but does not seem trivial is it is vital to the good fitting of the model. With the synthetic data, the exact number of homophilic factors was known (10 towns) and number of heterophilic were known (2 genders), leading to an over simplified problem, as this knoweldge is unknown for a real problem. Exporing this would strengthen the work, like showing the behaviour of the model if choosing 9 homophilic and 3 heterpholic features for example, or how to find those from the data maybe? Some more details about how many features and other parameter tuning of the comparison models in the results would also be important to understand if these seem optimal. 

clarity,_quality,_novelty_and_reproducibility: CL.

THe paper is very clearly written and presented. A few small questions I did have though to help clear up some details.

Fig. 1 - can you add some details on the sampling? Not critical though.

Could you elaborate on why kB + kC = 6k is required?

If I understand correctly, homophilic clusters, in B, are found as those with positive eigenvalues and heterophilic clusters, in C, those with negative eignevalues?

How to choose the kB and kC parameter values? Can these eigenvalues above somehow help with this, although for large data this probably not computationally feasible, can you comment?

QU.

"Many graph models fall under the framework of edge-independent graph generative models" - some references, including a review paper would be valueable here to support this claim.

In Figure 3 the learnt features are shown, and the proposed method has captured the ten cities and the additional 2 gender features. Big Clam only learns the ten cities; have you tried with 20 features and BigClam, as this has the potential to learn each combination of GENDER X CITY as an independent feature element?

NO.

There is a recent paper on overlapping community detection with embeddings that would fit into the cited work here, however, they do not address heterophily as far as I understand, nor do they show proof of exact representation but still it is close to this work:

Li, Y., Sha, C., Huang, X., & Zhang, Y. (2018). Community Detection in Attributed Graphs: An Embedding Approach. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.11274

The contirbution of introdcuing the heterophilic factors (C) is a very nice contribution, and when this certainly gives the model the capability to fit a graph, with homophilic and heterophilic relations between nodes, with an interpretable representation. A weakness here though is the difficulty to choose the number of positive and negative factors for algorithm, an interesting topic. Additionally, if the other methods, like BigClam have more features, like 20 in the synthetic example, are they able to learn as well given the extra capacity? If so then this method is surely faster, and more interpretable but maybe the ability to reduce the frobenious norm error will be smaller?

RE.

Code is included in the supplementary material. Without the package versions, it may be very hard to get this running in the future. Adding file versions as text in the README or a requirements.txt can solve this problem, including the Python version.


summary_of_the_review: A very good model, with a new lower bound and a simple optimisation approach. The achilles heel seems to be selecting the number of homophilic factors and number of heterophilic factors. This is not explored in the experiments nor discussed in the paper. Clearing this up would make a strong paper.

[Post Review Updates]

My questions were clearly answered and I feel that this is a novel and sginificant contribution to a growing are of interest: working with heterophily in graphs.

I have raised my review to an accept as I see this work as ready for publication.

[Post Reviewer-Discussion]
During the reviewer discussion the similarity to the work on Generalized RDPG was brought up and this area of work is not compared in enough detail considering the similarity with the work on homopholic and heterophilic relations in [1]. There was also some questions about the bound, to be clarified. Therefore I have slightly lowered my review recommendation as I felt these were genuine concerns.

I want to emphasize that in my opinion this is an area of growing interest, namely modelling heterophilic and homophilic relations. I feel this paper is well written and shows promising results. I believe this will be a very strong paper with the additions of addressing more closely the GRDPG and related literature, clarifying questions on the bound and ensuring that the real-data experiments are strongly supporting the claims (datasets are those which have heterophilic and homophilic relations, and the comparable methods are state-of-the-art).

1. P. Rubin-Delanchy, J. Cape, M. Tang, and C. E. Priebe, "A statistical interpretation of spectral embedding: The generalized random dot product graph," 2017

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: Yes, Discrimination / bias / fairness concerns
details_of_ethics_concerns: Minor potential issue is using the heterosexual dating example for the synthetic data. 

I would reconsider the example of only hetergoenous dating sites just as it could be seen as exclusive, there can of course be bisexual people on those sites and maybe also homosexual people on those sites and maybe it's not so nice to base the example on assuming there is no interest in the male-male or female-female relations. The example is very clean though and logical and I sadly fail to think of a good alternative. Is there maybe an example in the medical or chemistry area where there is a simlar repulsion between connected elements. How about an example with fraudseters and customers like in this review [Zheng et al. 2022]? Could it even be enough to just say that in a same-gender dating site the example would actually be homophilic between genders so we use a hetersexual example? A small issue, but just in case it could offend.

Zheng, Xin, et al. "Graph neural networks for graphs with heterophily: A survey." arXiv preprint arXiv:2202.07082 (2022).

recommendation: 6: marginally above the acceptance threshold