From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: notable-top-25%
metareview:_summary,_strengths_and_weaknesses: This work proposes a novel method for differentiable hypergeometric distributions over subsets.

The reviewers and I agree that this is a very good submission that presents new, interesting ideas, in a well-exposed manner.
justification_for_why_not_higher_score: see meta-review
justification_for_why_not_lower_score: see meta-review
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper6350/Reviewer_gdaG
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes an approximation of the multivariate non-central hypergeometric distribution in which, first the multivariate distribution is expressed through the product rule as a product of conditional distributions, each of which can be treated as a univariate non-central hypergeometric distribution by grouping the remaining (un-accommodated/sampled) classes into one. Finally, these univariate non-central hypergreometric distributions are approximated using the Gumbel-Softmax approximation technique. This makes the differentiation of the joint distribution with respect to the class importance weights feasible.

The paper concludes with some experiments; first showing the accuracy of the approximation through simulating from the true and approximate distributions, and then in two real applications where it shows superiority to some existing methods in learning numbers of shared and independent latent generative factors from coupled image observations and in clustering, where the proposed distribution is used as a tunable prior for a VAE.
strength_and_weaknesses: Strengths:
- The work investigates a relatively under-studied topic in machine learning where alternatives perform modelling based on i.i.d. sampling and may, at best, use post-hoc assessment of group sizes or similar to infer importance weights. By building the importance weights into the estimation both inference and estimation can be improved.
- The empirical results show that the method can offer improvements on existing methods in some interesting and relevant problems.

Weaknesses:
- The main weakness of the paper is that it expects a lot of knowledge from the reader, and many "for details see..." pieces of text which may make the work not stand well along as a single item of literature.
- There are also potential issues with clarity, where while the individual points are clear it is not immediately forthcoming how the optimisation rendered possible by the differentiability of the mass function is implemented.
clarity,_quality,_novelty_and_reproducibility: For the most part the paper is very readable and most individual points are reasonably clear, however, as hinted above the paper could benefit from an extremely simple application in which the optimisation which is rendered possible by the proposal is described explicitly.  Having said this, the actual description of the algorithm, and the sampling procedure for generating realisations from the distribution is clear and, having some experience with other machinery described in the experiments I believe the results are reproducible.
summary_of_the_review: The paper covers an interesting and under-explored topic in machine learning, and the proposed method seems to offer options for modelling, estimation and inference which are otherwise quite limited to heuristics. Although the paper is very readable and the details are well explained, as a reader one might feel still somewhat at a loss for some of the higher level practical aspects of the proposed approximation.
 The experiments show that the proposed distribution has realisable benefits in some important and interesting applications over existing alternatives and more naive straightforward formulations.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper6350/Reviewer_uGmy
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper proposes a differentiable approximation of the hypergeometric distribution and shows that it works well.
strength_and_weaknesses: Strengths:
- Very clear contribution in form of a practical tool that can be used by others
- Potential for high impact, in a similar manner as the concrete/GS distribution had an impact on how categorical variables are used in broad range of models
- Very well written for a paper this technical

Weaknesses:
- Nothing really, but a graphical illustration of the hypergeometric distribution and the role of the augmentation would not hurt; it would be especially useful if this becomes a core reference for people not so familiar with the literature
clarity,_quality,_novelty_and_reproducibility: The paper is so well written and provides a comprehensive treatment on the matter with sufficient demonstrations of the practical value that I do not have any remarks concerning the content. The technical content is not terribly complicated and one could argue that this is a fairly natural continuation of the earlier relaxations, but nevertheless something that has not been done yet and the devil is often in the details that probably were not clear before all the effort that went to developing this. I commend the authors on being very clear on the limitations and even small technical details that will be important for practical implementations (e.g. clearly saying how log-domain makes calculations more stable etc).
summary_of_the_review: Well written comprehensive treatment of a practical tool that makes building models that rely on the hypergeometric distribution easier.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper6350/Reviewer_yMNr
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes an approximation of the hypergeometric distribution that introduces a differentiable hyperparameter. The benefit is that, in machine learning algorithms where data may fall into discrete categories, the number and size of the categories is potentially learnable with gradient methods. The paper illustrates this distribution numerically in two settings: learning the number of dependent components in a weak supervision setting and clustering.
strength_and_weaknesses: Strengths

* Seems like a potentially useful tool.


Weaknesses

* Methodology could still use further investigation. The Kolmogorov-Smirnov test is one nice step, but it would also be nice to mathematically quantify the difference between the hypergeometric and the proposed distribution with standard measures of distributional similarity (various divergences, TV distance, etc.).


Questions

* It seems like this methodology is most applicable for variational inference, which combines gradient optimization and distributional inference. Are there other methodological applications I should have in mind?
clarity,_quality,_novelty_and_reproducibility: Clarity, Quality, Novelty, Reproducibility

* Clarity could be improved in the applications.
* The rest seem fine, although my read on novelty in this area is limited.

summary_of_the_review: Summary

* Potentially a useful methodology for working with distributions having underlying categories in some situations.


correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6350/Reviewer_xbJ8
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors propose a differentiable formulation of the hypergeometric distribution using the Gumbel-Softmax trick. They provide a sampling algorithm and demonstrate its use on two tasks that require modeling of discrete distributions. The first is a weakly-supervised representation learning problem for video frames. The second is a deep variational clustering application. In both applications, the use of the differentiable hypergeometric distribution improves performance. Additionally, the authors demonstrate that for a wide range of parameter values, one cannot distinguish between the a baseline non-differentiable implementation of the hypergeometric distribution and the proposed differentiable implementation using a KS test.

strength_and_weaknesses: Modeling categorical data using neural architectures has always been a challenge. The proposed method extends the tools available to model different classes of problems. Additionally, this is accomplished without significantly increasing the number of parameters.

My only concern with this work is the issue of fitting the importance weights. The authors do not seem to spend any time discussing the implications of this method on the convergence of gradient optimization routines. Can this method cause exploding or vanishing gradients? If so, under what conditions? Some remarks, analysis, or empirical evidence on this topic would be welcome.

clarity,_quality,_novelty_and_reproducibility: The manuscript is well written and clear.
While the approach is constructed of well known pieces; conditional sampling and the Gumbel-Softmax trick, they are used in a smooth and elegant fashion. As a consequence, I find the work novel and potentially impactful.

summary_of_the_review: The authors propose a useful tool for modeling discrete problems in differentiable frameworks.

correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper