From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: The paper extends the work of Zügner et al. (2021), called FPH, to developing a hierarchical clustering method. They extend the soft hierarchy scores of FPH to expected hierarchy scores, prove results about it and present experimental support for the method, showing small improvements over FPH.  This achieves a continuous approach to hierarchical clustering.  The approach is an improvement on the prior FPH work.

The paper is well written.  The presentation has good components, with theory, experiments and discussion.  The theory tightens up the earlier FPH work. 

The reviewers, however, felt the improvement over FPH was smallish and the contribution is really a cleanup of FPH rather than a major original contribution.   Thus the reviewers felt the work was not strong enough.




justification_for_why_not_higher_score: The authors did a great job of addressing the reviewers concerns. 
justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: N/A

From: ICLR.cc/2023/Conference/Paper4872/Authors
title: Open concerns
comment: Dear Reviewer,

We are happy to answer and address any unclarities or remaining concerns. If you have further questions, please let us know.

Best regards.

From: ICLR.cc/2023/Conference/Paper4872/Authors
title: Open concerns
comment: Dear Reviewer,

We are happy to answer and address any unclarities or remaining concerns. If you have further questions, please let us know.

Best regards.

From: ICLR.cc/2023/Conference/Paper4872/Authors
title: Author's response
comment: We are happy that our new comparison addressed the reviewer's concern. 

We would also like to mention that all other reviewers' concerns have been addressed point by point. Furthermore, we would like to again highlight the novelty of our submission: This paper introduces the *new* EPH model which has *new* theoretical properties and consistently achieves the best results in most vector and graph datasets. To scale it to large vector datasets a *new unbiased* Dasgupta approximation is proposed (see "Contributions"). We are happy to tackle any remaining issues.

From: ICLR.cc/2023/Conference/Paper4872/Reviewer_Cazv
title: Response to Authors
comment: Thank the authors for their responses and new comparison. 

I am happy with the new comparison. However, other reviewers also pointed out some other issues with the paper. I think the results of this paper are interesting, but the techniques used in this paper are not very novel or strong enough. So, I still think it is quite marginal and tend to reject it if other reviewers have no comments.

From: ICLR.cc/2023/Conference/Paper4872/Authors
title: Author's response
comment: We are happy that our answer addressed the reviewer's concerns.

We would like to emphasize that we do *not* use the training scores of EPH for the experimental evaluation. While EPH optimize the *expected* Dasgupta and TSD scores, the experimental evaluation uses the *discrete* Dasgupta and TSD scores which are different in general (see "Discrete vs Expected metrics" in answer to reviewer Cazv). In particular, the expected scores operate over relaxed probabilistic hierarchies, while the discrete scores operate over discrete hierarchies. This is similar to the baselines gHHC, UFit, HypHC and FPH which also propose to optimize different relaxed versions of the Dasgupta and/or TSD scores during optimizations. Further, the average linkage and the recursive sparsest cut have performance guarantees (Moseley & Wang, 2017) with respect to the Dasgupta cost, thus we include them. Hence, the baselines are evaluated in a fair way.

Furthermore, this experimental evaluation is well-established. Indeed, previous works (Monath et al., 2017, Chierchia & Perret, 2019, Chami et al., 2020, Zugner et al., 2021) use the discrete Dasgupta and/or TSD scores which can be computed unsupervised, and capture multi-scale information of a hierarchy. In contrast, while the accuracy is an appealing indirect application of hierarchical clustering, it can only be computed based on a set off labels representing a single scale of information.

References: 

- Chami, Ines, Albert Gu, Vaggos Chatziafratis, and Christopher Ré. "From trees to continuous embeddings and back: Hyperbolic hierarchical clustering." Advances in Neural Information Processing Systems 33 (2020): 15065-15076.
- Chierchia, Giovanni, and Benjamin Perret. "Ultrametric fitting by gradient descent." Advances in neural information processing systems 32 (2019).
- Monath, Nicholas, Ari Kobren, Akshay Krishnamurthy, and Andrew McCallum. "Gradient-based hierarchical clustering." In 31st Conference on neural information processing systems (NIPS 2017), Long Beach, CA, USA. 2017.
 - Moseley, Benjamin, and Joshua Wang. "Approximation bounds for hierarchical clustering: Average linkage, bisecting k-means, and local search." Advances in neural information processing systems 30 (2017).
 - Zügner, Daniel, Bertrand Charpentier, Morgane Ayle, Sascha Geringer, and Stephan Günnemann. "End-to-End Learning of Probabilistic Hierarchies on Graphs." In International Conference on Learning Representations. 2021.

From: ICLR.cc/2023/Conference/Paper4872/Reviewer_9qMA
title: empirical comparison based on training scores
comment: Thank you for addressing my concerns. 

Regarding point 5 (on the empirical comparison), the point is not whether there is a risk of "data leakage" from labels, as you put it, but whether it is fair to compare the training scores of your method with the scores of baselines that do not use this evaluation metric as objective. This issue is also raised by reviewer Cazv. 
The problem of this comparison (i.e., based on the training objective of EPH) becomes clear from the results of Table 3. The fact that EPH has better training Dasgupta cost does not imply that the learned trees are better (as assessed by their accuracy), i.e. training Dasgupta cost is not predictive of the quality of the trees learned by EPH.

From: ICLR.cc/2023/Conference/Paper4872/Authors
title: Author's response
comment: We would like to thank the reviewer for the detailed and helpful feedback. We updated the manuscript based on the feedback with important text modifications highlighted in blue. We are happy to move some new results from the appendix into the main text if the reviewers do not object, and are happy to clarify any points that remain unclear. Further, as mentioned in the reproducibility statement, we will make our model & training code publicly available after acceptance.

1. **Clarification on optimum**: By the term *optimum* we were referring to the *optimal value* (i.e. the optimal score values) achieved by an *optimizer* (i.e., an optimal hierarchy), of which there can be multiple. Note that, since the expectation operator is convex, any *discrete optimizer* (i.e. discrete hierarchies achieving the optimum value) of the discrete scores will be an optimizer of the expected scores and vice-versa. In this case discrete hierarchies are represented by deterministic $A$, $B$ matrices. Only probabilistic hierarchies which are optmizers of the expected scores, represented by non-discrete $A$, $B$ matrices, are not optimizers of the discrete scores. This is expected since those probalistic hierarchies do not belong to the valid input domain of the discrete scores. In addition, any sample we draw from these probabilistic optimizers is also a discrete optimizer of Dasgupta or TSD because of the convexity of the expectation operator. To avoid confusion, we have replaced the term "optimum" by "optimal value" where appropriate, and we further improved the clarity of these points in the updated paper in A.5 as well as Sec. 4.2.
5. **Clarifications on unbiased estimates**: While the estimation of the Dasgupta cost is unbiased, the backward pass is indeed biased as previously mentioned in "Limitations" in Sec. 4.3. We followed the reviewer's suggestion and make this point more clear in the udpated paper in "Model contribution" in Sec. 1 and "Limitations" in Sec. 4.3.
6. **Validity of sampled hierarchical structure**: The validity of the sampled hierarchy structure is guaranted by the *tree-sampling procedure* (see "Probabilistic hierarchical clustering" in Sec. 3). In the samples obtained by the tree-sampling procedure, each (leaf or internal) node is assigned exactly one parent node which is higher in the hierarchy, leading to a valid tree structure. More precisely, the parent of the leaf node $i$ is sampled from $Cat(A_{i:})$ and the parent of the internal node $i'$ is sampled from $Cat(B_{i':})$ where B is triangular.
7. **Clarification Eq. 3**: The optimization problems in Eq. 3 optimize over the matrices $A$ and $B$, which parametrize the probabilistic hierarchy, while the edge weights are given by the dataset. We clarify this point in the updated paper below Eq. 4 in Sec. 4.1.
8. **Clarification Figure 2**: The unweighted $K_4$ graph has 6 edges, thus every edge has a normalized edge weight of $\frac{1}{6}$. We added this for clarification into the caption. While we agree that the hierarchy in b) might not be intuitively the best, it is the best in terms of Dasgupta cost. This is only a simple example to illustrate that FPH sometimes fails to find the optimum (in terms of Dasgupta cost) while EPH does.
9. **Training/testing splits, Validation Score, & Std deviations**: Since we are in an unsupervised setting, i.e. the Dasgupta cost and TSD scores are computed without any labels, we train and evaluate on the whole dataset without risking data leakage. We clarify this in Sec. 5.1 in Datasets. The results for FPH and EPH are the scores from the discrete hierarchies which we obtain after the training. We add the standard deviations in Tab. 16 and Tab. 17. To compute the accuracies FPH and EPH are trained on Soft-Das and Exp-Das, since TSD is a metric for graphs.

From: ICLR.cc/2023/Conference/Paper4872/Authors
title: Author's response
comment: We would like to thank the reviewer for the valuable feedback. We updated the manuscript based on the feedback with important text modifications highlighted in blue. We are happy to move some new results from the appendix into the main text if the reviewers do not object.

1. **Discrete vs Expected metrics**: The optimization problems of the discrete and expected metrics are different in multiple ways. The discrete metrics operate over *discrete* hierarchies (see Eq. 3) while the expected metrics operate over *probabilistic* hierarchies (see Eq. 4). Contrary to the expected metrics, discrete scores optimization is often computationally expensive to solve and is not amenable to differentiable optimization (see Sec. 1 below Eq. 1). Hence while their optimal values are equal (see Theorem 1), discrete and expected metrics have important different practical properties.
2. **Iterative sparsest cut**: Thank you for suggesting another baseline method. Based on this, we add the recursive sparsest cut (RSC) to our baselines. We implement the algorithm based on (Dasgupta, 2016). The results are added to Tab. 2 and Tab. 3. The standard deviations are shown in Tab. 16 and Tab. 17 and an ablation on the number of internal nodes in Fig. 13 and Fig. 14.
3. **Improvement significance**: While the absolute improvements sometimes appear small, we would like to highlight that the improvements are quite consistent (EPH achieves the best results for 20/24 datasets w.r.t. Dasgupta and TSD scores), and improvements go up to, e.g., 12% (relative) improvement on OpenFlight.

References:
- Dasgupta, Sanjoy. "A cost function for similarity-based hierarchical clustering." In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pp. 118-127. 2016.

From: ICLR.cc/2023/Conference/Paper4872/Authors
title: Author's response (2/2)
comment: - **Clarification Figure 3**: We updated the Fig. 3 for clarity. The previous figure was using the minimum Dasgupta cost among 5 seeds, which is valid for an unsupervised task but is more senstive to outliers. Instead, we now show the mean. It shows that more sampled hierarchies lead to better convergence. Additionally, we followed the reviewer suggestion and provided the std errors in Fig. 15. Note that Fig. 3 shows the *result of training* five models with the respective number of sampled hierarchies as a hyperparameter setting. Thus, the fluctuations are not guaranteed to become smaller as we increase the number of samples. We additionally include another figure which shows the *result of estimating* the expected Dasgupta cost as we increase the number of samples (see Fig. 16 in B.6). As expected we see a convergence for increasing numbers of samples.

References:
- Chami, Ines, Albert Gu, Vaggos Chatziafratis, and Christopher Ré. "From trees to continuous embeddings and back: Hyperbolic hierarchical clustering." Advances in Neural Information Processing Systems 33 (2020): 15065-15076.
- Charpentier, Bertrand, and Thomas Bonald. "Tree sampling divergence: an information-theoretic metric for hierarchical graph clustering." In IJCAI-19. 2019.
- Dasgupta, Sanjoy. "A cost function for similarity-based hierarchical clustering." In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pp. 118-127. 2016.
- Moseley, Benjamin, and Joshua Wang. "Approximation bounds for hierarchical clustering: Average linkage, bisecting k-means, and local search." Advances in neural information processing systems 30 (2017).
- Zügner, Daniel, Bertrand Charpentier, Morgane Ayle, Sascha Geringer, and Stephan Günnemann. "End-to-End Learning of Probabilistic Hierarchies on Graphs." In International Conference on Learning Representations. 2021.

From: ICLR.cc/2023/Conference/Paper4872/Authors
title: Author's response (1/2)
comment: We thank the reviewer for the helpful and constructive comments. We updated the manuscript based on the feedback with important text modifications highlighted in blue. We are happy to move some new results from the appendix into the main text if the reviewers do not object.

- **Motivation for Dasgupta/TSD**: We use Dasgupta and TSD scores since they are both quality metrics for hierarchical clustering wiht intuitive motivations. Dasgupta is a well-established metric for graphs and vector data (Dasgupta, 2016, Chierchia & Perret, 2019, Chami et al., 2020, Zugner et al., 2021). Intuitively, it favors similar leaves to have lowest common ancestors low in the hierarchy (Dasgupta, 2016). TSD is a recent metric specifically designed for graphs. Intuitively, TSD quantifies the ability to reconstruct the graph from the hierarchy in terms of information loss (Charpentier & Bonald (2019)). We made this point more clear in the updated version in Sec. 4.1.
- **EPH vs FPH**: The main distinction between EPH and FPH is the optimization problem that they consider. EPH optimize the expected dagupata and TSD scores which can be written $E_{\hat{T}}[Das(\hat{T})]$ and $E_{\hat{T}}[TSD(\hat{T})]$ and are estimated with MC sampling. FPH optimizes the soft Dasgupta and TSD scores which can be written $Das(E_{\hat{T}}[\hat{T}])$ and $TSD(E_{\hat{T}}[\hat{T}])$ and are computed with closed-form formula. We clarify this point in the udpated paper in sec. 4.1 below Eq. 3. 
- **Clarification Theorem 1**: A and B are assumed to be row stochastic meaning that a full row cannot be zero. Further, the tree-sampling procedure guarantees to sample valid and non-disconnected hierarchies (see "Probabilistic hierarchical clustering" in sec. 3). Indeed, the parent of the leaf node $i$ is sampled from $Cat(A_{i:})$ and the parent of the internal node $i'$ is sampled from $Cat(B_{i':})$. Thus, each node has exactly one parent with the last internal nodes connecting to the root. In practice, $A$ and $B$ are initialized with the average linkage algorithm (see "Experimental Setup" in sec. 5.1) and then optimized with projected gradient descent which guarentees them to be row stochastic (see "Constrained vs Unconstrained Optimization" in App. B.5). We make this point more clear in the updated paper in "Experimental Setup" in Sec. 5.1. The edge weights $w_{ij}$ are given by the dataset and used to compute the node and edge distribution. For graphs, we use the edge weights in the data (or simply all ones in the case of unweighted graphs), and for vector data we use the cosine similarity of the feature vectors (see "Datasets" in Sec. 5.1). We clarified this in the updated version after Eq. 4 and added a formal description of the normalization procedure in App. B.8.
- **Explanation of EPH steps**: EPH is composed of three main steps represented on Fig. 1: the "Differentiable Hierarchy Sampling" (see Sec. 4.3), the "Subgraph Sampling" (see Sec. 4.4), and the "Loss Computation" (see Sec. 4.4). Beyond the previous reference to Fig. 1 at the end of Sec. 4, we added new references to Fig. 1 in Sec. 4.3. and Sec. 4.4 to clarify the EPH steps. Finally, we followed the reviewer suggestion and also added the formal EPH algorithm in App. B.8.
- **Clarification Experiments**: In general, linkages algorithms are strong baselines. In particular, average linkage also has theoretical guarantees in terms of Dasgupta cost (Moseley & Wang, 2017). We follow the experimental setup in (Zugner et al., 2021) regarding the choice of internal nodes and DeepWalk dimensions. While a higher number of internal nodes allows more expressive hierarchies, we observed that Dasgupta and TSD scores plateau at around 512 (see new Fig. 13 and Fig. 14). We added a clarification paragraph on the number of internal nodes in Sec. B.5. 
- **Runtime**: We added the runtimes of different approaches in Tab. 16 and Tab. 17. The runtime of EPH depends on the number of sampled hierarchies and the number of sampled edges in the subgraph-sampling step (see end of Sec. 4). Note that HypHC is executed 45 times since we perform a small hyperparameter search (see Tab. 9 in App. B.2). Further, since gHHC is substantially faster than the other randomized methods, we ran it for 45 additional random seeds for fairness, and updated the results tables (Tab. 2 and Tab. 3) accordingly.

From: ICLR.cc/2023/Conference/Paper4872/Reviewer_EPYj
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors consider the problem of inferring a hierarchal clustering of dataset instances.  The problem has typically been approached as either a discrete optimization problem or a continuous, relaxed form of the discrete problem.  In the discrete setting one searches over all possible hierarchies - with each instance having a single internal parent node and each internal node (other than the root) having a single internal parent node.  In the continuous setting, nodes are softly assigned to more than one parent node.  The authors focus on probabilistic hierarchical clustering, which maintains a distribution over discrete hierarchies and introduce Expected Probabilistic Hierarchies (EPH) as a means to optimize two hierarchical clustering cost metrics - Dasgupta cost and Tree-Sampling Divergence - in expectation.  They show that maximizing these two cost methods in expectation is consistent with the non-probabilistic variant, introduce a method for efficiently sampling hierarchies as the number of instances grows and demonstrate the effectiveness of the method on some real datasets and synthetic datasets.
strength_and_weaknesses: Overall, I thought the paper was well written and had a clear and natural development.  The main idea is a logical extension of prior work, but to the best of my knowledge is novel and is a non-trivial extension. The main theoretical results are intuitive and provide a nice justification for the proposed sampling procedure.

While the paper was well written, I do think the authors could have devoted more time to explaining the reasons for using the Dasgupta cost and TSD?  Is it because prior art already existed for them?  Also, I am not an expert in probabilistic hierarchical clustering methods and the distinction between EHC and Zugner et al.’s FPH method was unclear.  In addition, I would liked to have seen a more formal description of the proposed algorithm, including the subgraph sampling procedure.  EPH seems primarily like a way to model distributions over hierarchies under different cost metrics. The reader is left to infer the steps needed to actually optimize over this model.

clarity,_quality,_novelty_and_reproducibility: I am a bit confused by a few aspects of Theorem 1.  Does the equality make some assumption about the matrices A and B?  It seems to me if these matrices had zeros, then not every tree structure could be sampled.  In an extreme case, the zeros could cause the induced hierarchy to be disconnected.  Along these lines, where do the initial values for A and B come from in practice?  In the experiments section it seems as if a uniform transition distribution is assumed.  Similarly, where do the edge weights w_ij connection nodes i and j come from? For a dataset of n instances, do we need to provide (and normalize) over all n choose two pairs of instances?

Figure 1 did not provide much value.  I don’t think it was referenced or explained in the paper and the caption did not provide any context or explanation. As I said above, I really would like there to be a formal algorithm description.  In its current presentation, there are too many gaps for me to reasonably recreate the experiments.  

I found the experiments section a bit difficult to follow.  I appreciate that the authors wanted to be exhaustive in their comparison, but the discrete single, average complete and ward linkage based approaches did not seem like reasonable methods to compare with in the main results section.  They are heuristic and don’t try to optimize the Dasgupta or TSD metrics. Can you provide a justification for why “we use n′ = 512 internal nodes, compress hierarchies using the scheme presented by Charpentier & 
Bonald (2019), and use 10 and 32-dimensional DeepWalk embeddings (Perozzi et al., 2014) on the graphs for methods that require features”?  These steps were quite unclear to me and I am not able to discern their effect on the experimental results.  Why did you choose n’=512?  

The tables seem to show that EPH finds low cost hierarchies.  Can you comment on the run time needed for the different methods?  If the EPH takes 10 hours, but the other methods take only 5 minutes, then I’d argue the comparison should be based on computation time rather than fixing 5 random seeds.  

Figure 3 is very noisy.  What’s happening with the citeseer dataset?  Why doesn’t score decrease as function of increased number of samples? Also, are there error bars on the markers in Fig 3?  I would hope the variance in dasgupta costs decreases as num sampled hierarchies is increased too.
summary_of_the_review: Overall, I like this paper.  The idea of sampling over a distribution of hierarchies is appealing and I think the authors did a nice job of illustrating how to use the probabilistic model to optimize over the dasgupta cost and TSD.  However, I’m not sure if the paper in its current form is quite good enough.  With additional clarity of the procedure and of the theoeretical result, it could be good enough.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4872/Reviewer_Cazv
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper considers hierarchical clustering with expected objectives.  This problem aims to optimize the expected objective (e.g. Dasgupta cost or Tree-sampling divergence) over a distribution of discrete hierarchies. They show that this problem has the same global objective as its discrete counterpart.  They propose a new method called the expected probabilistic hierarchies (EPH), which is an end-to-end gradient-descent-based optimization by using a differentiable hierarchy sampling. They also scale this method to large graphs with an unbiased subgraph sampling approach. They show that their algorithm performs well on real-world datasets.
strength_and_weaknesses: Strengths:
1. They propose an expected optimization problem over hierarchies.
2. They propose a new end-to-end learning framework for this problem.
3. Their method outperforms baselines on real-world datasets. 

Weaknesses:
1. The proposed two expected optimization problems seem identical to their discrete counterparts.
2. How would this new method compare with the method based on iterative sparsest cuts for Dasgupta's cost?
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to understand. The method in this paper is original and reproducible. However, the techniques used in this paper are more or less standard and similar to previous works. 
summary_of_the_review: The problem considered in this paper is essentially the same as its discrete counterpart. They propose a new end-to-end learning framework for this problem while the techniques are quite standard. Their method achieve better performance than baselines while the improvement is very marginal and the objectives of some baselines are different from the evaluation objective.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper4872/Reviewer_9qMA
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The paper builds on a recent modelling of the hierarchical clustering of [Zugner et al. 2021] to learn discrete structures by gradient descent.
The structure is modelled using two row-stochastic parameter matrices: $A$ parametrizes the probability distributions of each sample of belonging to a cluster ($n$ distributions); $B$, a strictly upper triangular matrix, parametrizes the hierarchy between clusters, i.e. the probability of a cluster of being a child of another cluster up in the ordering ($n'-1$ distributions).

While [Zugner et al. 2021] optimizes soft-version of the objectives Das and TDS, the current work proposes to optimize a Monte Carlo approximation of the expected objectives using a perturb-and-MAP sampling scheme, more precisely leveraging the gumbel-softmax trick.
Because in the considered parametrization each probability distribution is an independent categorical, this optimization procedure is unbiased in the space of continuous hierarchies. 

The method is finally applied on graph and vector data, and shown to outperform existing methods when optimizing the metric used for evaluation (not when evaluating accuracy).
strength_and_weaknesses: ### Strengths
The paper is very well-written: the motivation is convincing and the contributions are stated clearly. The theoretical and model contributions are somehow novel, as they correspond to a novel application (i.e., hierarchical clustering) of well-known techniques and results in discrete optimization.

### Weaknesses
1. Several times in the paper (e.g., abstract, introduction) it is claimed that **"the optimum of the expected scores is equal to the optimum of their discrete counterparts"**. This statement **is not accurate**: the optimal score is the same (and this is what is proved in the appendix) but the optima are the same only in the case that the optimum of the discrete score is unique (i.e., not degenerate).
Furthermore, **this claim coupled with the statement that "EPH provides an unbiased estimate of Exp-Das and Exp-TSD" misleads the reader to think that the overall optimization is unbiased w.r.t. the discrete hierarchy**, which is not true (see discussion in [1] and [2]). Although these claims are partially rectified in Section 4.3, under Limitations, I encourage the authors to clarify all these points from the beginning.

2. Regarding the chosen parametrization of the hierarchies from [Zugner et al. 2021], I have a concern about the modelling of the cluster hierarchy by B.  Because it is row-stochastic, the only constraint enforced is that the probabilities of a non-root node of being a child of the other nodes sum to 1. This means that non-hierarchical structures are allowed in this parametrization. 
To give an example, suppose we have 3 clusters: the root $z_3$, and $z_1$, $z_2$ internal nodes. if z_2 belongs to z_3 and z_1 belongs to z_2, with the chosen parametrization $z_1$ can also belong to z_3. Could the authors elaborate on this point?

3. In Eq. 3 it would be helpful to specify which variables are given by the task at hand and which are optimized. While reading the paper, it is not clear whether the node and edge probabilities are determined by the dataset (e.g., by the similarities). This confusion stems from the existence of two different graphs in the problem formulation: the undirected graph given by the dataset and the learned directed acyclic graph for the hierarchy. It would help the reader to stress this difference and use different terms for the edges and nodes of the dataset graph.

4. In Figure 2, without the similarities between nodes in the $K_4$ graph it is hard to appreciate that hierarchies (b) and (d) model the problem better than (c). As all the nodes are connected to all the others, hierarchy (c) seems on the contrary more suited as it assigns all nodes to the same cluster. Moreover, without the similarities the scores given in the caption come out of the blue. 

5. In the empirical evaluation, it is not clear whether the datasets are split into training/test samples. It is important to clarify this point in order to appreciate the results of Table 2 and 3. Are the scores reported for EPH and FPH the training losses? Also, which objective is used when reporting accuracy: Das or TSD? In general, the standard deviations of the results should be also reported, in order to assess the significance of the gaps.

### Minors
- page 4: "of which $z$ is an ancestor of" -> whose ancestor is $z$
- page 7: hierarchical -> hierarchy

[1] Chris J. Maddison, Andriy Mnih, Yee Whye Teh: The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. ICLR, 2017

[2] Shakir Mohamed, Mihaela Rosca, Michael Figurnov, Andriy Mnih: Monte Carlo Gradient Estimation in Machine Learning. J. Mach. Learn. Res., 2020

clarity,_quality,_novelty_and_reproducibility: **Several points have to be clarified**, concerning the theoretical contribution and the empirical analysis (see Weaknesses). In particular, two inaccurate claims and missing details on the experimentation affect the quality of the paper.

**The contributions have limited novelty** in that they constitute a novel application of well-known results in discrete optimization leveraging a recently proposed probabilistic formulation of hierarchical clustering. 

The implementation and the code for running the experiments were not submitted. The paper do not report all the details for reproducing the results, but this can be easily fixed.

summary_of_the_review: The paper has several weaknesses, that I believe should be easy to fix. The most concerning one is the inaccuracy on the unbiasedness of the learning procedure.
Otherwise, the paper is well-written. The contributions have limited novelty, but could have a good impact in the hierarchical clustering domain.

**update after discussion**

The authors addressed the concerns I raised and revised the paper accordingly. Because of this, I raised my score to 6.
I am not willing to increase it more as the work boils down to an application of well-known techniques in discrete optimization that (i) did not require deriving any new results and (ii) in practice does not provide better structures (see Table 3 - accuracy).
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold