Paper:
Learning a Concept Hierarchy from Multi-labeled Documents
While topic models can discover patterns of word usage in large corpora, it is difficult to meld this unsupervised structure with noisy, human-provided labels, especially when the label space is large. In this paper, we present a model-Label to Hierarchy (L2H)-that can induce a hierarchy of user-generated labels and the topics associated with those labels from a set of multi-labeled documents. The model is robust enough to account for missing labels from untrained, disparate annotators and provide an interpretable summary of an otherwise unwieldy label set. We show empirically the effectiveness of L2H in predicting held-out words and labels for unseen documents.

Review:
Given a corpus of multi-labeled documents, the author(s) propose a topic model which associates each label with a topic, and recovers a tree-structured hierarchical taxonomy of the topics/labels.  The model is evaluated on a political science data set.  As well as finding such interpretable structure, as seen in qualitative results in the paper, the model is reported to improve perplexity over LDA and labeled LDA, and also provides features which outperform TF-IDF and labeled LDA at multi-label document classification.Using label information to create a hierarchy of topics and labels is a nice idea, bringing together separate strands of work on hierarchies of topics and hierarchies of labels.  The proposed model is elegant and is a principled approach to the problem.  An approximation must be made in the Gibbs sampling procedure (Equation 7), though it is unclear how accurate the approximation is.  Although the procedure of label selection using TF-IDF when performing document classification seems somewhat ad-hoc, to be fair it is a reasonable approach, and the strategy has been used before by Ramage et al.However, unfortunately, the procedure used to estimate the perplexity scores for Equation 9 is incorrect.  In the following, for simplicity of exposition, I will describe the issue for standard LDA instead of for the proposed model, ignoring the variables which are specific to this model such as the labels. The notation here also explicitly conditions on \alpha.In the manuscript, for each MCMC sample of the topics, the procedure seems to be to draw Z from its posterior given each test document.  This procedure fits Z to the *test* data.  It would therefore over-estimate the likelihood of data which are truly unobserved, and would reward over-fitting of Z.In more detail, the procedure in the manuscript estimates \sum_z P(W|Z, \phi, \alpha) P(Z|W, \phi, \alpha).  However, what should be estimated for Equation 9 is, for each sample of \phi, the likelihood for the test document, P(W|\phi, \alpha) = \sum_z P(W,Z|\phi, \alpha) = \sum_z P(W|Z, \phi) P(Z|\alpha).  In other words, the present procedure computes an average over the *posterior*.  However, the likelihood is equal to an average over the *prior*.To correct this, an unbiased estimate for the likelihood, to plug into Equation 9, can be computed using these samples from the posterior, by instead taking the harmonic mean instead of the arithmetic mean (Equation 15 in Wallach et al. 2009).  However, Wallach et al. found that this method has extremely high variance, and hence is not recommended.  They suggest alternative algorithms such as the left-to-right method and annealed importance sampling.Another sensible alternative strategy is to repeat the exact procedure used in the manuscript, except first splitting each test document into two portions, "test_train" and "test_test."  The procedure in the paper (sample Z from the posterior given the data and topics) is performed on "test_train," and the Rao-Blackwellized estimates of theta from the resulting Z's are used to evaluate perplexity on "test_test," thus computing P(test_test|test_train, \phi).  This is the "estimated theta document completion" strategy, described in Section 5.1 of Wallach et al. (2009).  For more information on evaluating topic models, see:Wallach, H. M., Murray, I., Salakhutdinov, R., & Mimno, D. (2009). "Evaluation methods for topic models." Proceedings of the 26th Annual International Conference on Machine Learning.Going forward, it would be good if the author(s) could comment on the semantics of the extracted hierarchy.  Clearly the model does not always place strictly more specific topics/labels below more general topics/labels, although the generative model encourages this somewhat.  So if the tree structure does not always encode generality relationships, how should we interpret it?   It would also be good to present results on more than one data set to give more evidence that the results hold generally.*** Thanks to the authors for addressing my concern regarding the evaluation procedure.  I have adjusted my score accordingly.