Paper:
Supervised Word Mover's Distance
Recently, a new document metric called the word mover's distance (WMD) has been proposed with unprecedented results on kNN-based document classification. The WMD elevates high-quality word embeddings to a document metric by formulating the distance between two documents as an optimal transport problem between the embedded words. However, the document distances are entirely unsupervised and lack a mechanism to incorporate supervision when available. In this paper we propose an efficient technique to learn a supervised metric, which we call the Supervised-WMD (S-WMD) metric. The supervised training minimizes the stochastic leave-one-out nearest neighbor classification error on a perdocument level by updating an affine transformation of the underlying word embedding space and a word-imporance weight vector. As the gradient of the original WMD distance would result in an inefficient nested optimization problem, we provide an arbitrarily close approximation that results in a practical and efficient update rule. We evaluate S-WMD on eight real-world text classification tasks on which it consistently outperforms almost all of our 26 competitive baselines.

Review:
This paper proposes a supervised metric learning approach built on top of the recent “word-movers distance” [19] (a transport distance applied to word2vec word embeddings). The proposed approach can be applied to any transport distance based on squared Euclidian distances after linear projection. On top of learning the linear projection to maximise performance, the proposed approach also learns a weighting of words.  The latter is specific to the document application, it consists in re-weighting the frequency of each word by a learned weight, and re-normalising the word frequencies. The paper presents an efficient optimisation method to learn the proposed parameterised metric. A thorough experimental evaluation shows the effectiveness of the proposed approach by comparing it to a large number of reasonable baselines on a number of document classification datasets using a kNN classifier based on the proposed learned metric. 
+ This work extends the recent word-movers distance, which is an interesting technically and useful extension (shown to be effective on a number of kNN text classification tasks).   - The potential impact is perhaps limited by the fact that earth-movers distance with a metric learning via a “word centroid distance” initialisation performs similar to the full learning of the proposed distance. See the difference of the last two rows in Table 3. In other words, the main technical contributions of this paper do not seem to contribute very much over this initialisation method based on existing techniques. - One thing that I missed in table 3 is an evaluation of the “word centroid embedding” (weighted mean of word embeddings for the document) with all tested existing metric learning approaches (ITML, LMNN, NCA). - Another point missing in the evaluation is an assessment of the relative importance of the word-specific weights, and the projection matrix A. It would be useful to see performance with each one of these components being active in isolation.  o The originality is mostly related to the way to optimise the metric (Line 156–175), the idea of a linear metric learning within the earth-movers distance is very interesting but a limited contribution in itself.   + The paper is very well written and a pleasure to read. There are a few minor issues that could be clarified: - Line 152: “The authors”: of which paper? [1,7,8] ? - Equation 11: shouldn’t alpha and beta carry stars here ? - It would be useful to explicitly recall around line 170 that this is the Sinkhorn scaling that is referred to in the introduction in line 56.