Paper:
Semi-crowdsourced Clustering with Deep Generative Models
We consider the semi-supervised clustering problem where crowdsourcing provides noisy information about the pairwise comparisons on a small subset of data, i.e., whether a sample pair is in the same cluster. We propose a new approach that includes a deep generative model (DGM) to characterize low-level features of the data, and a statistical relational model for noisy pairwise annotations on its subset. The two parts share the latent variables. To make the model automatically trade-off between its complexity and fitting data, we also develop its fully Bayesian variant. The challenge of inference is addressed by fast (natural-gradient) stochastic variational inference algorithms, where we effectively combine variational message passing for the relational part and amortized learning of the DGM under a unified framework. Empirical results on synthetic and real-world datasets show that our model outperforms previous crowdsourced clustering methods.

Review:
Post Rebuttal Summary -- Thanks to the authors for a careful rebuttal that corrected several notation issues and provided an attempt at more realistic experiment using CIFAR-10 with real (not simulated) crowd labels. I found this rebuttal satisfactory and I am willing to vote for acceptance. I won't push *too* hard for acceptance because I still wish both the new experiment and the revised presentation of variational method details could go through more careful review.  My chief remaining concerns are that it is still difficult to distinguish the reasons for gains from BayesSCDC over plain old SCDC because multiple things change (factorization of q(z,x), estimation of global parameters). I do hope the authors keep their promise to compare to a "middle of the road" 3rd version and also offer insight about how amortization makes the q(z|o)q(x|z,o) approach still less flexible than mean field methods.  Review Summary -- Overall, I think the core ideas here are interesting, but I'm rating this paper as borderline accept because it doesn't do any experiments on real crowd-labeled datasets, misses opportunities to provide take-home insights about the various variational approximations attempted here, suffers from some minor bugs in the math (nothing showstopping), and leaves open several procedural questions about experiments. My final decision will depend on a satisfactory rebuttal.   Paper Summary -- This paper considers the problem where a given observed dataset consists of low-level feature vectors for many examples as well as pair-wise annotations for some example pairs indicating if they should belong to the same cluster or not. The goal is to determine a hard/soft clustering of the provided examples. The annotations may be *noisy*, e.g. from a not-totally-reliable set of crowd workers.  The paper assumes hierarchical model for both observed feature vectors O and observed pair-wise annotations L. Both rely on a shared K-component mixture model, which assigns each example into one of K clusters. Given this cluster assignment, the O and L are then generated by separate paths: observed vectors O come from a Gaussian-then-deep-Gaussian, and pairwise annotations L are produced by a two-coin Dawid-Skene model, originally proposed in [16]. Many entries of L may not be observed.  Two methods for training the proposed model are developed: * "SCDC" estimates an approximate posterior for each example's cluster assignment z and latent vector x  which conditions on observed features O -- q(z, x | O) -- but point estimates the "global" parameters (e.g. GMM parameters \pi, \mu, \Sigma) * "BayesSCDC" estimates an approximate posterior for all parameters that aren't neural net weights, using a natural-gradient formulation.  Notably, for "SCDC" the posterior over z,x has conditional structure:     q(z,x|o) = q(z|o) q(x|z,o). In contrast, the same joint posterior under the "BayesSCDC" model has mean-field structure with no conditioning on o: q(z,x) = q(z)q(x).  Experiments focus on demonstrating several benefits: * 5.1 toy data experiments show benefits of including pairwise annotations in the model. * 5.2 compares on a Faces dataset the presented approach to several baselines that can also incorporate pairwise annotations to discover clusters * 5.3 compares the two presented methods (full BayesSCDC vs SCDC) on MNIST data   Strengths -- * The pair-wise annotation model nicely captures individual worker reliability  into a probabilistic model.  * Experiments seem to explore a variety of reasonable baselines.   Weaknesses --  * Although the method is intended for noise crowd-labeling, none of the experiments actually includes truly crowd-labeled annotations. Instead, all labels are simulated as if from the true two-coin model, so it is difficult to understand how the model might perform on data actually generated by human labelers.  * The claimed difference between the fully-Bayesian inference of BayesSCDC and the point-estimation of global parameters in SCDC seems questionable to me... without showing the results of multiple independent runs and breaking down the differences more finely to separate q(z,x) issues from global parameter issues, its tough to be sure that this difference isn't due to poor random initialization, the q(z,x) difference, or other confounding issues.    Originality --  The key novelty claimed in this paper seems to be the shared mixture model architecture used to explain both observed features (via a deep Gaussian model) and observed noisy pairwise annotations (via a principled hierarchical model from [16]). While I have not seen this exact modeling combination before, the components themselves are relatively well understood.  The inference techniques used, while cutting edge, are used more or less in an "out-of-the-box" fashion by intelligently combining ideas from recent papers. For example, the recognition networks for non-conjugate potentials in BayesSCDC come from Johnson et al. NIPS 2016 [12], or the amortized inference approach to SCDC with marginalization over discrete local variables from Kingma, Mohamed, Rezende, and Welling [13].  Overall, I'm willing to rate this as just original enough for NIPS, because of the technical effort required to make all these work in harmony and the potential for applications. However, I felt like the paper had a chance to offer more compelling insight about why some approaches to variational methods work better than others, and that would have really made it feel more original.  Significance -- The usage of noisy annotations to guide unsupervised modeling is of significant interest to many in the NIPS community, so I expect this paper will be reasonably well-received, at least by folks interested in clustering with side information. I think the biggest barriers to widespread understanding and adoption of this work would be the lack of real crowd-sourced data (all experiments use simulated noisy pairwise annotations) and helping readers understand exactly why the BayesSCDC approach is better than SCDC alone when so much changes between the two methods.    Quality Issues --  ## Q1) Correctness issues in pair-wise likelihood in Eq. 2  In the pair-wise model definition in Sec. 2.2, a few things are unclear, potentially wrong: * The 1/2 exponent is just wrong as a poor post-hoc correction to the symmetry issue. It doesn't result in a valid distribution over L (e.g. that integrates to unity over the support of all binary matrices). A better correction in Eq. 2 would be to restrict the sum to those pairs (i,j) that satisify i < j (assuming no self edges allowed).  * Are self-edges allowed? That is, is L_11 or L_22 a valid entry? The sum over pairs i,j in Eq. 2 suggests so, but I think logically self-edges should maybe be forbidden.  ## Q2) Correctness issue in formula for mini-batch unbiased estimator of pair-wise likelihood  In lines 120-122, given a minibatch of S annotations, the L_rel term is computed by reweighting a minibatch-specific sum by a scale factor N_a / |S|, so that the term has similar magnitude as the full dataset. However, the N_a term as given is incorrect. It should count the total number of non-null observations in L. Instead, as written it counts the total number of positive entries in L.  ## Q3) Differences between SCDC and BayesSCDC are confusing, perhaps useful to breakdown more finely  The presented two approximation approaches, SCDC and BayesSCDC, seem to differ on several axes, so any performance difference is hard to attribute to one change. First, SCDC assumes a more flexible q(x, z) distribution, while BayesSCDC assumes a mean-field q(x)q(z) with a recognition network for a surrogate potential for p(o|x). Second, SCDC treats the global GMM parameters as point estimates, while BayesSCDC infers a q(\mu, \Sigma) and q(\pi). I think these two axes should be explored independently. In particular, I suggest presenting 3 versions of the method: * the current "SCDC" method * the current "BayesSCDC" method * a version which does q(x)q(z) with a recognition network for a surrogate potential for p(o|x) (Eq. 10), but point estimates global parameters.  The new 3rd version should enjoy the fast properties of BayesSCDC (each objective evaluation doesn't require marginalizing over all z values), but be more similar to SCDC.   Clarity ------- The paper reads reasonably well. The biggest issue in clarity is that some some key hyperparameters required to reproduce experiments are just not provided (e.g. the Dirichlet concentration for prior on \pi, the Wishart hyperparameters, etc.). These absolutely need to be in a final version.   Other reproducibility concerns: * what is the procedure for model initialization? * how many initializations of each method are allowed? how do you choose the best/worst? accuracy on training?