Paper:
Efficient Loss-Based Decoding on Graphs for Extreme Classification
In extreme classification problems, learning algorithms are required to map instances to labels from an extremely large label set. We build on a recent extreme classification framework with logarithmic time and space [19], and on a general approach for error correcting output coding (ECOC) with loss-based decoding [1], and introduce a flexible and efficient approach accompanied by theoretical bounds. Our framework employs output codes induced by graphs, for which we show how to perform efficient loss-based decoding to potentially improve accuracy. In addition, our framework offers a tradeoff between accuracy, model size and prediction time. We show how to find the sweet spot of this tradeoff using only the training data. Our experimental study demonstrates the validity of our assumptions and claims, and shows that our method is competitive with state-of-the-art algorithms.

Review:
1. Summary  The paper concerns a problem of extreme classification. The authors provide an in-depth analysis of the recently proposed LTLS algorithm. They reformulate the problem in the framework of loss-based decoding for ECOC. Based on their theoretical insights they introduce valid extensions to the original algorithm. They discuss the trade-off between the computational and predictive performance, which is then validated experimentally. The introduced W-LTLS obtains promising results in the experiments.  2. Relation to previous work  The authors build their work on the LTLS algorithm and the loss-based decoding for ECOC. Both are very nicely introduced in a formal way at the beginning of the paper. The authors also include a section with a more general overview of related work.   Below I give only one minor remark in this regard. Since the authors also discuss the tree-based methods in the related work, the following algorithms (with logarithmic complexity)  should be mentioned: - Hierarchical softmax ("Hierarchical probabilistic neural network language model", AISTATS 2005) and FastText ("Bag of tricks for efficient text classification", CoRR 2016) - Conditional probability estimation trees ("Conditional probability tree estimation analysis and algorithms", UAI 2009) - Classification Probability Trees (the generalization of the above) ("Consistency of probabilistic classifier trees", ECMLPKDD 2016) - Filter trees ("Error- correcting tournaments", ALT 2009) - Probabilistic label trees ("Extreme F-measure maximization using sparse probability estimates", ICML 2016)  3. Quality:  3.1 Strengths  This is a solid submission that extends previous work. The new algorithm has been analyses from theoretical and empirical point of view. The authors also try to establish in a formal way the trade-off between predictive accuracy and computational complexity. This is one of the first steps into a general reduction framework for this kind of trade-off.  4. Clarity  4.1 Strengths  The paper is well and clearly written. The appendix contains almost all necessary elements for which there were no place in the main text.  4.2 Weaknesses  The empirical results should also be given in tabular form with numeric values, not only in the form of plots. This is important for accurate comparison of algorithms in the future references.  5. Originality:  5.1 Strengths  The algorithm and the theoretical results are original.  6. Significance:  6.1 Strengths  The work is significant as it will push the research in the desired direction of establishing the framework for the trade-off between predictive and computation performance.  7. Minor remarks:  References: Reference 1 has been doubled, please check also other references as they do not contain all relevant information.  8. After rebuttal  I thank the authors for their rebuttal. I do not change my evaluation, but give one additional comment that could be useful for the authors.  As the paper compares W-LTLS to LomTree and FastXML, it could also include the comparison to a label tree approach being either represented by HSM, PLT, or Parabel. All these algorithms are very similar to each other. PLT and Parabel are multi-label generalizations of HSM. Parabel additionally uses a specific tree building strategy that gets very good results for multi-label problems. To compress the model size in label tree approaches one can use a simple shallow network as it is done in the FastText implementation.