Paper:
A no-regret generalization of hierarchical softmax to extreme multi-label classification
Extreme multi-label classification (XMLC) is a problem of tagging an instance with a small subset of relevant labels chosen from an extremely large pool of possible labels. Large label spaces can be efficiently handled by organizing labels as a tree, like in the hierarchical softmax (HSM) approach commonly used for multi-class problems. In this paper, we investigate probabilistic label trees (PLTs) that have been recently devised for tackling XMLC problems. We show that PLTs are a no-regret multi-label generalization of HSM when precision@k is used as a model evaluation metric. Critically, we prove that pick-one-label heuristic-a reduction technique from multi-label to multi-class that is routinely used along with HSM-is not consistent in general. We also show that our implementation of PLTs, referred to as EXTREMETEXT (XT), obtains significantly better results than HSM with the pick-one-label heuristic and XML-CNN, a deep network specifically designed for XMLC problems. Moreover, XT is competitive to many state-of-the-art approaches in terms of statistical performance, model size and prediction time which makes it amenable to deploy in an online system.

Review:
Summary: This work investigates Probabilistic Label Trees (PLTs) in solving extreme multi-label classification (XMLC). The theoretical analysis shows PLT is a no-regret algorithm for precision@k, and the algorithmic improvement combines PLT and fastText to efficiently handle  extreme multi-label text classification problems, with a clustering-based tree structure building strategy.   This paper is comphrensive and well-written, including extensive experiments.  The theory part formally shows PLT outputing k labels with highest marginal probabilities is consistent with precision@k, given zero-regret node classifiers. The authors also provide some negative result on heuristic strategies, one is that pick-one-label heuristic is suboptimal in terms of precision@k, and another is that building Huffman trees for PLT does not minimize computational cost.  I think the consistency result is a bit straightforward since the zero-error assumption on marginal probabilities is strong enough to provide many nice properties. However, as the authors said in Line 231-233, if this assumption holds, there is no need to care about the tree structure in PLT, which is surely an important issue in practice. Also, it is easy to see that PLT is preferred than vanilla HSM in XMLC.  Personally speaking, I would like to see the authors put more theory parts into the supplementary and show more experiments on the tree structure decision in PLT. For example:  - The performance difference between original PLT paper vs. Huffman tree vs. top-down hierarchical clustering used in this paper. - The improvement by applying fastText into PLT. Does fastText help both in time cost and performance?  ============================== Post-author feedback comments:  I have read the author feedback and found out that their theoretical results are based on the maximum estimation error of marginal probability on nodes. Although this measure may not be available in practice, the theoretical contribution is more than what I expected before.   Therefore I increased my score.