Threshold optimization for F measure of macroaveraged precision and recall Abstract   There are two different approaches to macroaveraging F measure for multilabel classification. The first encloses averaging F measure over all classes, which makes it easy to optimize. The second, extensively investigated in this paper, comprises the F measure of macro precision and recall calculation. We examine and compare these two measures when applied to different multilabel datasets. To optimize the performance measure, we adopt a widely known and proven modular approach. Classifiers sort the instances in descending order, according to a realvalued score of belonging to a corresponding class. After that, thresholds are selected so as to optimize the performance measure. If the number of classes is sufficiently large and the second alternative of macroaveraging F measure is employed, then it becomes nontrivial to define the optimal number of instances to assign to each class. Cyclic optimization procedure is widely used for threshold optimization although it results in a maximum in a special coordinatewise sense. For a micro averaged F measure, such a coordinatewise optimum is a maximum in the conventional sense of this term but it is not true for the F measure of macro precision and recall, which is shown by a counterexample. We reduce the problem of selecting the optimal threshold for each class to the problem of obtaining a fixed point of a specifically introduced transformation of a unit square. The suggested algorithm lets us localize all possible coordinatewise maximums and detect the optimal among them. The approach is applied to datasets from diverse application domains.

Early Forecasting of Text Classification Accuracy and FMeasure with Active Learning. When creating text classification systems, one of the major bottlenecks is the annotation of training data. Active learning has been proposed to address this bottleneck using stopping methods to minimize the cost of data annotation. An important capability for improving the utility of stopping methods is to effectively forecast the performance of the text classification models. Forecasting can be done through the use of logarithmic models regressed on some portion of the data as learning is progressing. A critical unexplored question is what portion of the data is needed for accurate forecasting. There is a tension, where it is desirable to use less data so that the forecast can be made earlier, which is more useful, versus it being desirable to use more data, so that the forecast can be more accurate. We find that when using active learning it is even more important to generate forecasts earlier so as to make them more useful and not waste annotation effort. We investigate the difference in forecasting difficulty when using accuracy and Fmeasure as the text classification system performance metrics and we find that Fmeasure is more difficult to forecast. We conduct experiments on seven text classification datasets in different semantic domains with different characteristics and with three different base machine learning algorithms. We find that forecasting is easiest for decision tree learning, moderate for Support Vector Machines, and most difficult for neural networks.