Paper:
Consistent Binary Classification with Generalized Performance Metrics
Performance metrics for binary classification are designed to capture tradeoffs between four fundamental population quantities: true positives, false positives, true negatives and false negatives. Despite significant interest from theoretical and applied communities, little is known about either optimal classifiers or consistent algorithms for optimizing binary classification performance metrics beyond a few special cases. We consider a fairly large family of performance metrics given by ratios of linear combinations of the four fundamental population quantities. This family includes many well known binary classification metrics such as classification accuracy, AM measure, F-measure and the Jaccard similarity coefficient as special cases. Our analysis identifies the optimal classifiers as the sign of the thresholded conditional probability of the positive class, with a performance metric-dependent threshold. The optimal threshold can be constructed using simple plug-in estimators when the performance metric is a linear combination of the population quantities, but alternative techniques are required for the general case. We propose two algorithms for estimating the optimal classifiers, and prove their statistical consistency. Both algorithms are straightforward modifications of standard approaches to address the key challenge of optimal threshold selection, thus are simple to implement in practice. The first algorithm combines a plug-in estimate of the conditional probability of the positive class with optimal threshold selection. The second algorithm leverages recent work on calibrated asymmetric surrogate losses to construct candidate classifiers. We present empirical comparisons between these algorithms on benchmark datasets.

Review:
The paper provides a study and accompanying algorithms for binary problems where performance metrics go beyond the mere classification accuracy; such performances are F-measure, AM-measure, Jaccard similarity coefficient, and, of course, precision and recall measures. The findings of the authors are a few:a) they show that there are many performance measures that can be expressed as ratio of linear combinations of the false positive (FP) rates, true positive rates (TP), true negatives (TN) and false negative (FN) rates. Working out a bit these ratio, it turns out that they can be expressed with respect to TP and probability for the classifier to predict a +1 class.b) a consequence of a) is that it is possible to show the Bayes classifier (which optimizes the performance metric at hand) is simply of the form sign(eta_x - \delta) where eta_x is the conditional probability for Y=1 given x.c) 2 consistent learning algorithms building upon b) (and therefore a)) are provided and proof of consistency (of one of them) are provided (and correct, as far as I have checked)d) numerical simulations on both controlled and benchmark datasets are provided that illustrate the soundness of the algorithms. Finally, a few directions for future research such as multiclass classification and multilabel classification are outlined.Quality------The paper is of very good quality, both from the mathematical and writing viewpoints. It makes reference to the pivotal works in the same family of contributions that help understand the proposed contributions. The structure of the paper is very sound and there is a nice balance between the depiction of basic things (cf. the linear ratios), the mathematical part (on consistency), the algorithmic part and the experimental part.Clarity------The paper is very clear. The authors made a good choice in postponing the most technical part to the appendix.Originality---------It is very nice to discover that there is still very interesting (and not just technicall) work to do in the realm of binary classification. This is all the more so that the topic addressed (richer performance measure than the accuracy) are of very high practical interest.Significance----------It is no doubt that this paper will be of very high interest for people working in information retrieval or for those looking for new learning framework to chew from the statistical standpoint.Minor comments--------------- insert a section 2.1 (named, e.g. "Metrics as ratios of linear combinations...") before the current section 2.1, or remove section title 2.1. (If there is only one subsection then there is no need to have any numbering.)- l. 387: "the methods, the methods"