Paper:
On the Statistical Consistency of Plug-in Classifiers for Non-decomposable Performance Measures
We study consistency properties of algorithms for non-decomposable performance measures that cannot be expressed as a sum of losses on individual data points, such as the F-measure used in text retrieval and several other performance measures used in class imbalanced settings. While there has been much work on designing algorithms for such performance measures, there is limited understanding of the theoretical properties of these algorithms. Recently, Ye et al. (2012) showed consistency results for two algorithms that optimize the F-measure, but their results apply only to an idealized setting, where precise knowledge of the underlying probability distribution (in the form of the 'true' posterior class probability) is available to a learning algorithm. In this work, we consider plug-in algorithms that learn a classifier by applying an empirically determined threshold to a suitable 'estimate' of the class probability, and provide a general methodology to show consistency of these methods for any non-decomposable measure that can be expressed as a continuous function of true positive rate (TPR) and true negative rate (TNR), and for which the Bayes optimal classifier is the class probability function thresholded suitably. We use this template to derive consistency results for plug-in algorithms for the F-measure and for the geometric mean of TPR and precision; to our knowledge, these are the first such results for these measures. In addition, for continuous distributions, we show consistency of plug-in algorithms for any performance measure that is a continuous and monotonically increasing function of TPR and TNR. Experimental results confirm our theoretical findings.

Review:
Quality: Technical quality is very good, the theoretical results are sound and all the proofs are provided either in the main body of the paper, or in the appendix.Clarity: Despite a strong emphasis on theoretical results (numerous lemmas and theorems), the paper was actually relatively easy to follow and the quality of writing is very good. Nevertheless, I think the paper is very heavy on notation, and this should be improved. I give some suggestions below, but I leave it to the authors what changes (if any) they prefer to include. For instance, I would suggest to drop subscript D in the whole paper, i.e. TPR_D, TNR_D, P_D, regret_D, etc., would be TPR, TNR, P, regret, etc. It obscures the notational, while the distribution is always the same (D) and clear from the context. It will also be easy to distinguish expected quantities from empirical quantities, as the latter are always indicated by the "hat" symbol. Moreover, it is probably not needed to explicitly denote the dependence on S_1 and S_2 all the time (e.g., by using \widehat{\eta}_{S_1} or \widehat{t}_{S_2,\widehat{\eta}_{S_1},\Phi}). In the same way, \Psi does not need to be used in almost all symbols, as it is always clear what \Psi do we mean (there is only one \Psi analyzed throughout the whole proof). I understand the authors wanted to be precise with notation, but I think it is obscure at the moment and has a negative effect on the readability of the paper.Originality: As far as I am concerned, the results are novel and generalize some previous results shown for very specific cases (specific measure \Psi, stronger assumptions).Significance: Judging by interesting theoretical results and good experimental verification, I think the paper should have an impact on the theory of learning with non-decomposable performance measures.In my opinion, this is a good paper with interesting theoretical result. Although \Psi-consistency of L_1-consistent class probability estimator is not really that surprising, this is the first formal result of such generality for non-decomposable performance measures, and is worth publishing. In my opinion, the paper would benefit from answering to at least some of the questions below.- Is assumption that the optimal threshold exists and is in (0,1) necessary? If yes, it would be nice to give an illustrative counterexample, which shows why consistency fails when the optimal threshold is in {0,1}, or when it does not exist. If not, it would be worth trying to prove the theorems without this assumption.- Is Assumption A necessary (as compared to B) for non-monotonic performance measures? Again, I am missing an example of what can go wrong if \Psi is not monotonically increasing in TPR and TNR, while Assumption A is satisfied and Assumption B is not. If no such counterexample can be given, again, the authors should try to relax Assumption A.- It would also be interesting to consider the case when probability estimates and threshold are obtained on the same data set S (without splitting into S_1 and S_2). This is a common learning scenario in the real life, and hence it is legitimate to ask under what conditions consistency holds in such a setting?Some specific remarks:- Line 185-186: what does it mean "eta(x) conditioned on y=1 and on y=-1 is mixed"?- Line 192 (Theorem 1): "for some t^*_{D,\eta,\Psi} \in (0,1)" -- this symbol (t^*_{D,\eta,\Psi}) is already defined (what does the word "some" mean here), and it follows from Assumption A, that it is in (0,1), so this statement is unnecessary (I think).