Early Stopping Based on Unlabeled Samples in Text Classification (2022)

Abstract: Early stopping, which is widely used to prevent overfitting, is generally based on a separate validation set. However, in low resource settings, validation-based stopping can be risky because a small validation set may not be sufficiently representative, and the reduction in the number of samples by validation split may result in insufficient samples for training. In this study, we propose an early stopping method that uses unlabeled samples. The proposed method is based on confidence and class distribution similarities. To further improve the performance, we present a calibration method to better estimate the class distribution of the unlabeled samples. The proposed method is advantageous because it does not require a separate validation set and provides a better stopping point by using a large unlabeled set. Extensive experiments are conducted on five text classification datasets and several stop-methods are compared. Our results show that the proposed model even performs better than using an additional validation set as well as the existing stop-methods, in both balanced and imbalanced data settings. Our code is available at https://github.com/DMCB-GIST/BUS-stop .

## introduction early stopping, a form of regularization, is a widely used technique to prevent a model from over-fitting @xcite . it is generally based on a separate validation set @xcite . while monitoring the validation performance during training, the training process stops when the validation error starts to increase. validation-based early stopping is advantageous because it is easy to implement and can be interpreted directly @xcite . in a scenario where sufficient labeled data are available, the use of a validation set is generally preferred @xcite . however, when * corresponding author only a few labeled data exist, a tradeoff problem is encountered @xcite . for example, although the usage of a relatively large validation set enables more reliable estimation, the number of samples for training becomes insufficient. conversely, if small fractions of the samples are assigned to the validation set, the stopping point becomes ambiguous because the small validation set is not representative enough. early stopping is more important in a low resource setting because the prediction accuracy fluctuates highly during training. such high fluctuations render it challenging when to stop the model. one way to mitigate these fluctuations is to use sufficient training data. in this context, training all the available samples would be more effective, and for this purpose, an appropriate stopping point should be determined without validation split. however, this has not been extensively studied. @xcite and @xcite proposed gradient-based stop-methods and applied statistical inference on the training samples. @xcite suggested the usage of local intrinsic dimensionality (lid) for early stopping. in addition, some studies treat the stopping epoch as a hyperparameter: the stopping epoch is obtained by grid-search or averaging in cross validation @xcite . these methods allow the training of all the labeled samples. however, they do not consider the task-related performance metrics (e.g., accuracy) during training, and the lid and gradient-based stop criteria have not been commonly used in natural language processing (nlp). furthermore, gradient-based stop-criteria depend on the training samples, the size of which may still be small to be representative. in this study, we propose an early stopping method based on unlabeled samples (bus-stop). we are motivated by the following two considerations: (i) the probabilities of the predicted class label (i.e., the prediction confidences) can serve as an indicator for over-fitting or under-fitting. (ii) in a better model, the output class distribution is more likely to be closer to the class distribution of the true labels. to incorporate these two assumptions, two stop criteria are proposed, and combined in the bus-stop method. our method monitors the prediction results of unlabeled samples during training and utilizes them for determining the stop-criteria. the first proposed stop-criterion is based on confidence similarity (conf-sim). the model stops when the prediction confidences for the unlabeled samples are most similar to the reference confidences, which are precalculated on the labeled set with cross-validation. conf-sim is observed to reflect the long-term trend of the loss curve, and thereby assist in preventing over-training. the second stop criterion is based on the class distribution similarity (class-sim). this criterion stops the model when the predicted class distribution on the unlabeled set is most similar to the pre-estimated distribution. to this end, we present a novel estimation method for the true class distribution, which calibrates the predicted distribution by extrapolation such that it is closer to the true distribution. classsim is observed to reflect the short-term trend of the accuracy. our method requires several retraining steps to obtain the reference confidences for confsim and the estimated class distribution for classsim. the bus-stop method that combines classsim and conf-sim includes the advantages of both, and thereby performs with better accuracy and loss compared to each (class-sim and conf-sim). the following characteristics of our method contribute to performance improvement. our method does not require a separate validation set; hence, all the labeled samples can be trained. training can stop at a more generalized model, using a large unlabeled set. the proposed stop-criteria, conf-sim and class-sim, consider two performance metrics, namely, the loss and accuracy. our contributions are summarized as follows: ## related work prechelt (1998) experimented on 14 different validation-based stop criteria. @xcite focused on an issue that the validation error during training may represent many local minima prior to a global optimum. existing non-validation stop-criteria are generally based on statistical inference. @xcite interpreted stochastic gradient descent in terms of the variational inference and proposed an estimation method for the marginal likelihood of the posterior, which was applied as an early stopping criterion. however, this method requires considerable computation for the hessian, which is not practical in large models. @xcite also proposed a gradient-related stopping method referred to as evidence-based stopping (eb). the eb-criterion is based on the fast-to-compute local statistics of the computed gradients. the criterion represents whether the gradients of the training samples lie within the expected range. intrinsic dimensionality (id), which refers to the minimum number of parameters required to represent a dataset, has been used for analyzing the training or redundancy of neural networks @xcite . lid is a version of id that estimates the subspace dimensions of the local regions. @xcite found that lid works well as a stopping-criterion in several few-shot image classification datasets. moreover, lid can be applied to unlabeled samples. another method involves the pre-estimation of the the number of training epochs by training the model multiple times, such as cross validation @xcite ; the model can stop at the pre-estimated (pe) stop-epoch when training all the labeled samples. however, these methods have not been commonly studied for nlp tasks and do not consider the performance metrics during training. furthermore, comparisons among the non-validation stopmethods have not been reported. in this study, we compare our method with the eb, lid, pe, and validation-based stopping methods on five text classification datasets. the method proposed by algorithm 1 preliminary stage for bus-stop initialize a model, m split d l into d train and d val at a ratio of r train the m with (d train , d val ) m ← load the m that was the best on @xcite was not compared because it involves considerable computational cost. ## method in this section, we describe the proposed method in detail. the main notations used are as follows: ## discussion impact of the training size figure 3 indicates the accuracy curve with respect to the training size, using the imdb dataset. the x values of valstop add(x) and val-stop split(x) were set to 25, 25, 50, 100, 200, and 400, according to the increase in k. it can be observed that the performance of bus-stop is good in the sufficient-data regime as well. however, the performances of the three stop-criteria converge almost similarly with the increase in the training size. the impact of splitting the samples for validation does not deteriorate the performance when k is greater than 400. rather, val-stop split(x) performs slightly better when k is 1600. this result suggests that when sufficient labeled data are available, validation-based stopping can be a better choice. ## conclusion and future work validation-based early stopping can be detrimental in low-resource settings because the reduction in the number of samples by validation split may result in insufficient samples for training. in this study, we proposed an early stopping method called bus-stop, based on unlabeled samples. moreover, we proposed a calibration method to better estimate the true class distribution, which was used in the bus-stop method to improve the performance. we conducted experiments on five popular text classification datasets. the results indicated that bus-stop outperformed the existing stop-criteria in both balanced and imbalanced settings. in particular, busstop showed robustness to imbalanced classification. the proposed bus-stop method enables the training of all the available samples and presents a better stopping point using large unlabeled samples. in future, we plan to better exploit the unlabeled samples in self-training schemes. we can also combine bus-stop and self-training methods. bus-stop can be used to improve the performance of the initial model, which plays an important role in the final self-training performance. additionally, we consider applying the bus-stop to domain adaptation tasks in the future.