Early Forecasting of Text Classification Accuracy and FMeasure with Active Learning. When creating text classification systems, one of the major bottlenecks is the annotation of training data. Active learning has been proposed to address this bottleneck using stopping methods to minimize the cost of data annotation. An important capability for improving the utility of stopping methods is to effectively forecast the performance of the text classification models. Forecasting can be done through the use of logarithmic models regressed on some portion of the data as learning is progressing. A critical unexplored question is what portion of the data is needed for accurate forecasting. There is a tension, where it is desirable to use less data so that the forecast can be made earlier, which is more useful, versus it being desirable to use more data, so that the forecast can be more accurate. We find that when using active learning it is even more important to generate forecasts earlier so as to make them more useful and not waste annotation effort. We investigate the difference in forecasting difficulty when using accuracy and Fmeasure as the text classification system performance metrics and we find that Fmeasure is more difficult to forecast. We conduct experiments on seven text classification datasets in different semantic domains with different characteristics and with three different base machine learning algorithms. We find that forecasting is easiest for decision tree learning, moderate for Support Vector Machines, and most difficult for neural networks.

Learn: A Novel Incremental Learning Method for Text Classification Abstract   Deep learning is an effective method for extracting the underlying information in text. However, it performs better on closed datasets and is less effective in realworld scenarios for text classification. As the data is updated and the amount of data increases, the models need to be retrained, in what is often a long training process. Therefore, we propose a novel incremental learning strategy to solve these problems. Our method, called Learn, includes four components: a Student model, a reinforcement learning (RL) module, a Teacher model, and a discriminator model. The Student models first extract the features from the texts, then the RL module filters the results of multiple Student models. After that, the Teacher model reclassifies the filtered results to obtain the final texts category. To avoid increasing the Student models unlimitedly as the number of samples increases, the discriminator model is used to filter the Student models based on their similarity. The Learn method has the advantage of a shorter training time than the OneTime model, because it only needs to train a new Student model each time, without changing the existing Student models. Furthermore, it can also obtain feedback during application and tune the models parameters over time. Experiments on different datasets show that our method for text classification outperforms many traditional OneTime methods, reducing training time by nearly 80%.