Paper:
Sparse Local Embeddings for Extreme Multi-label Classification
The objective in extreme multi-label learning is to train a classifier that can automatically tag a novel data point with the most relevant subset of labels from an extremely large label set. Embedding based approaches attempt to make training and prediction tractable by assuming that the training label matrix is low-rank and reducing the effective number of labels by projecting the high dimensional label vectors onto a low dimensional linear subspace. Still, leading embedding approaches have been unable to deliver high prediction accuracies, or scale to large problems as the low rank assumption is violated in most real world applications. In this paper we develop the SLEEC classifier to address both limitations. The main technical contribution in SLEEC is a formulation for learning a small ensemble of local distance preserving embeddings which can accurately predict infrequently occurring (tail) labels. This allows SLEEC to break free of the traditional low-rank assumption and boost classification accuracy by learning embeddings which preserve pairwise distances between only the nearest label vectors. We conducted extensive experiments on several real-world, as well as benchmark data sets and compared our method against state-of-the-art methods for extreme multi-label classification. Experiments reveal that SLEEC can make significantly more accurate predictions then the state-of-the-art methods including both embedding-based (by as much as 35%) as well as tree-based (by as much as 6%) methods. SLEEC can also scale efficiently to data sets with a million labels which are beyond the pale of leading embedding methods.

Review:
The submission describes a scalable method for large multi-label prediction problems with very long-tailed label distributions.The method first learns a nonlinear embedding of the label (output) vectors that preserves local distances.A linear map is then learned from features (inputs) into this embedding space.At test time, the linear map is evaluated to predict a point in the embedding space, and then the labels are predicted via KNN in this space (i.e., the predicted label vector is the sum of those of training examples with similar embeddings).A few tricks are used to achieve scalability and good generalization: namely, clustering is used to reduce the complexity of KNN, and averaging over the randomness of clustering results is used to mitigate the instability of high-dimensional clustering.Experiments show that this method is faster to evaluate, more scalable at training, and generalizes better than competing methods.Significance: The experimental results are the strongest part of the paper, which show convincingly better results in multiple aspects compared to existing state-of-the-art methods.However, the method is a bit complex compared to other approaches, in the sense that it is a pipeline with a few steps and a number of parameters to tune.This makes it a bit less likely to catch on and have a big practical impact.Originality: The basic solution approach builds on previous embedding-based methods, and as such, the component parts have mostly appeared elsewhere previously.The most novel part about the method seems to be the idea to use a nonlinear, local-distance-preserving embedding as opposed to a linear, global embedding of the labels, but this is a fairly straightforward innovation.Clarity: Much can be done to improve the clarity of the submission.First, the paper suffers from information overload in a few places, making it difficult to tell what is important and what is not.As mentioned elsewhere, the theoretical analysis, though interesting, seems to be of little relevance to the actual method, so perhaps this section should be abridged.The experiments are very thorough, which is good, but in this section and throughout the paper, I would have preferred it if the discussion had focused on discussing the most important points well rather than trying to cram in as many points as possible.Quality: Overall, this is a well-thought-out method to solve a real-world problem well.Thoughtful concern is given to the many details required to make a system performant and scalable on real data.The experiments are also thorough with strong results.The theoretical analysis also seems very sophisticated, but unfortunately seems like a tangent with little relevance to the rest of the work.The weakest part of the work is that the novelty of the method is relatively low.Other comments:From a technical perspective, I was a bit disappointed at the disconnect between the proposed optimization for the embedding and the method used to solve it.The idea of learning an embedding that is nonlinear as a map from labels, but linear as a map from features is very interesting, because the linearity-from-features constraint serves to regularize an embedding which might otherwise be too powerful to produce meaningful results.Unfortunately, the algorithm used to actually optimize this objective learns the embedding completely ignoring the constraint that it must be realizable as a linear map from the features.So, the obtained solution effectively optimizes an objective which differs from the desired objective (3) in a very important way.This is also one reason why the ERM bound fails to provide justification for the method: there is little reason to believe that the obtained solution should be close to the optimum of (3).A related concern is that preserving local distances alone is not sufficient to ensure good accuracy of KNN, since a nonlinear embedding can easily preserve local distances while also pushing points into neighborhoods in which they do not belong.This is why methods such as Large Margin Nearest Neighbor exist.Unfortunately, such methods tend to be computationally expensive, so I worry that taking this into account would spoil the computational efficiency of the method.These concerns make it a little surprising that the method works so well in practice.POST-REBUTTAL COMMENTS:The rebuttal did not significantly change my opinion.I am positive overall about the work, mostly due to the experimental results, but I still have a few concerns.Basically, the presentation makes it seem as though certain decisions are motivated by rigorous theory, but this is not really the case.The two-step optimization procedure used in practice differs in a critically important way from the theoretically well-motivated joint optimization.The fact that the objective does not properly preserve neighborhoods also makes it surprising that the method works well in practice.If I were to revise the manuscript, I would put less emphasis on the marginally-related theoretical results and more emphasis on giving plausible explanations, empirical or otherwise, for these counterintuitive phenomena.