Paper:
Optimizing F-Measures by Cost-Sensitive Classification
We present a theoretical analysis of F -measures for binary, multiclass and multilabel classification. These performance measures are non-linear, but in many scenarios they are pseudo-linear functions of the per-class false negative/false positive rate. Based on this observation, we present a general reduction of Fmeasure maximization to cost-sensitive classification with unknown costs. We then propose an algorithm with provable guarantees to obtain an approximately optimal classifier for the F -measure by solving a series of cost-sensitive classification problems. The strength of our analysis is to be valid on any dataset and any class of classifiers, extending the existing theoretical results on F -measures, which are asymptotic in nature. We present numerical experiments to illustrate the relative importance of cost asymmetry and thresholding when learning linear classifiers on various F -measure optimization tasks.

Review:
The paper concerns a problem of training classifiers that maximize the F-measure. The authors show a reduction of the problem to cost-sensitive classification with unknown costs. They prove that such an approach is consistent and that in practical situations a good approximation can be obtained. The resulting algorithm follows an idea of training several cost-sensitive classifiers with different costs and choosing the one with the best F-measure. This is a quite different approach in comparison with the common solution in which a threshold maximizing the F-measure is tuned on a validation set. I like this paper, however, I have to admit that the structure of the paper could be improved. For example, I would rather prefer to present first the algorithm and its theoretical analysis for binary classification. Then, the authors could mention the extensions for other settings. I also tried to follow the proofs, but it was not always easy to do that. The authors should consider to move them to an appendix and present in a clearer way. The authors should also discuss wider the difference between the EUM (expected utility maximization) and DTA (decision-theoretic approach) frameworks, as these two are two different formulations of the problem. For example, the solution for DTA  does not necessarily converge to the solution of EUM for the micro F-measure.The introduced algorithm is very simple. Once we know that the optimal strategy is to find a threshold on conditional probabilities, we can either try to find the threshold on a validation set or use reduction to cost-sensitive classification with different costs (as in reduction of probability estimation to cost-sensitive binary classification). I would rather expect a more sophisticated strategy for finding the cost-sensitive problem with the best F-measure. Minor comments:- The sentence: "the optimal classifier (...) for the pseudo-linear F-measures is also an optimal classier for a cost-sensitive classification problem with label dependent costs" should be, in my opinion, stated in the other direction, i.e., "the optimal classifier for a cost-sensitive classification problem with label dependent costs is also optimal for the F-measure". The proof is in the right direction.- The authors should also present running times of the algorithms.After rebuttal:I thank the authors for their responses. I am looking forward for the revised version of the paper.