Curating Datasets for Better Performance with Example Training Dynamics (2023)

Abstract: The landscape of NLP research is dominated by large-scale models training on colossal datasets, relying on data quantity rather than quality. As an alternative to this landscape, we propose a method for weighing the relative importance of examples in a dataset based on their Example Training dynamics (swayamdipta et al., 2020) — a set of metrics computed during training. We propose a new way of computing the ETD of a dataset, and show that they can be used to improve performance in both in-distribution and out-of-distribution testing. We show that ETD can be transferable, i.e., they can be computed once and used for training different models, effectively reducing their computation cost. Finally, we suggest an active learning approach for computing ETD during training rather than as a preprocessing step — an approach that is not as effective, but dramatically reduces the extra computational costs.

## introduction breakthroughs in nlp are often the result of scaling up existing models in size and depth, and perhaps even more importantly-data @xcite . to improve data quality, it has become common practice to train models on data that has been cleaned to some extent using simple heuristics (e.g. by removing non-language tokens), but not otherwise optimized for better performance. while some data-filtering approaches have been suggested to improve out-of-distribution (ood) generalization @xcite , they usually result in a decrease of in-distribution (id) performance. we propose a method for curating datasets for better performance in both id and ood testing, enhancing data quality rather quantity. our method is orthogonal to model architecture or size, and as such can be used alongside any lm to further improve results over specific tasks. to implement our method we use the concept of example training dynamics (etd; @xcite , which builds on the idea that the training process of models sheds light on the relative importance of specific examples within the datasets used. specifically, @xcite have shown that over several epochs of training, a model may predict some examples in a dataset less consistently than others, and that those "ambiguous" examples are important for ood generalization. we propose a new method for computing etd, as well as a new paradigm for using them in training. we show that by computing etd over separate training processes (rather than over consecutive epochs of the same training process), and using etd to weigh the importance of each example in the dataset, we can train a deberta model @xcite on the weighted versions of several nli and multiple-choice datasets, improving average performance by 0.35% for id testing and by 0.95% for ood. we next demonstrate that etd can be transferable across models, i.e., etd computed from the training process of model m 1 can be used to weigh a dataset, and train model m 2 on it with improved results, where m 2 differs from m 1 in initial weights, structuring details and pre-training scheme. though we only show that for a specific use-case, if transferability of etd holds generally, it may allow us to create weighted versions of datasets once, and use them for multiple training scenarios, including that of future models. finally, we propose dynamic training, a method for training a model while computing etd and reweighing the training set between epochs. this method performs on par with our transferability method while requiring no additional compute beyond that of standard training, which makes it even more applicable under low computational budgets. our proposed method of computing etd may allow practitioners to get more value out of their existing datasets, and pave the way towards similar methods to be used for improving large scale language model training. we publicly release our code, as well the weighted versions of the datasets used in this work. foot_0 2 example training dynamics ## training with etd improves performance we follow @xcite and first test the capacity of our method to improve the performance of a model of the same architecture as the model used for computing the etd. we use six tasks, divided into three groups. three multiple @xcite ; and a question answering task: boolq @xcite . for each of the tested tasks, we compute etd using one copy of deberta-large @xcite . following swayamdipta et al. (2020) , we use @xmath0. we then train a second copy of deberta on the etd-weighted dataset d (.25,3) . the specific values of f and b are chosen using a grid search for the best mean performance over the development sets of all tasks. 3 due to computational constraints, we do not tune any hyperparameters other than those defined specifically for this work, i.e. f and b. for other hyperparameters such as learning-rate and batch-size we follow the values used in @xcite for training on the snli (bowman et al., 2015) dataset. for each task, we test the trained model on its designated test set to evaluate id performance, as well as on the test sets of all other tasks from the same group to evaluate ood performance (e.g., we evaluate a model trained on winogrande on αnli and hellaswag as well). as baselines, we train deberta on three additional datasets: these improvements are statistically significant: modelling the result of the baseline method in each category c as a sample from a normal distribution n c , the probability of outperforming the baseline when sampling from the same n c is 0.5. therefore, the probability of outperforming the baseline on all 14 tasks when sampling from their respective distributions can be calculated using a binomial random variable b(14, 0.5), which gives p-@xmath1, 0.5) = 14] = 0.00006. ## etd are transferable the process of computing etd is costly in terms of compute, requiring compute roughly equivalent to that of training the desired model. thus, computing the etd of a dataset separately for every model we wish to train is expensive, and, depending on the size of the dataset, may become prohibitively so. this problem can be bypassed if etd are transferable across models, i.e., if etd can be computed using a model m 1 , and then used to train a different model m 2 . to test for transferability, we use deberta as the etd-computing model, and electra @xcite as the training model. we conduct experiments similar to those in section 3, with the exception that the training model m 2 is electra. tables 3 and 4 show the results and summarized results of these experiments, respectively. when computing etd with deberta and creating the respective d (0.25,4) , training electra on d (0.25,4) outperforms training on d in 5 out of the 6 id categories, and in 9 out of all 14 categories. though not as consistent as our main method, the etd transfers well, with mean improvement of 0.2% id and 0.33% ood. ## from training dynamics to dynamic training we have established that example training dynamics can be computed as a pre-processing step and used to improve both id and ood performance. furthermore, thanks to tranferability it may be possible to save most of the compute involved in the process. but is it possible to go even further and use etd without any pre-processing? after all, any training process has its own naturally-occuring trainin dynamics. is it possible to compute and use etd within the same training process? with this question in mind, we propose dynamic training, an active learning method for weighing examples in datasets mid-training. dynamic training uses the same basic method as regular etdweighted training, except only one training process takes place, and the etd are being evaluated during training rather than as a preprocessing step. the etd are re-evaluated with each epoch of training, and thus the etd-weighted dataset d (f,b) on which the model trains changes between epochs. a dynamic training process starts with no etd whatsoever, so training is done on the unaltered dataset d. similar to computing regular etd, as training progresses, we save the model's probabilities for the labels of each example in d as etd. on some epoch e * > 1, we start using them to weigh d before each epoch. experiments to test the effectiveness of dynamic training , we use the method to train a de-berta model on the six datasets we used for our previous experiments. we use the hyperparameters e * = 3, @xmath2.33, @xmath3. we compare the results against a deberta model trained on d without any form of etd-weighted training. tables 5 and 6 show the results and summarized results of these experiments. dynamic training outperforms training on d in 5 out of the 6 id categories, and in 11 out of all 14 categories. though the anli task suffers decrease in performance, results for the other tasks improve relatively consistently, with mean improvement of 0.23% id and 0.38% ood. though not as effective as etd-weighted training, dynamic training improves performance in the majority of the cases, without any pre-processing of the data. ## robustness to hyperparameter tuning throughout this work we report results training models on etd-weighted datasets of the form d (f,b) , with specific values chosen for f, b. these values are hyperparameters, chosen using a grid search for best performance over the development set of the different tasks. ## related work previous research offered various methods that consider the relative importance of examples within a dataset in order to improve training. methods such as curriculum learning @xcite or self-paced learning @xcite focus on the order of training, advocating training on easy examples first (e.g., examples with high likelihood in the latter case). other methods rank the importance of examples with the goal of filtering datasets. @xcite use a train-twice approach with some resemblance to etd, and upweight train examples that were missclassified in the first round of training. aflite @xcite ranks examples based on the ability of a linear classifier to solve them, and then filters out easy examples in order to eliminate artifacts and biases from the dataset. other works advocated bias and artifacts removal, and by extension the removal of easy examples from datasets @xcite . core-set selection @xcite uses submodular functions to find a subset of a dataset representative of the whole, to be used under low computational budget. conversely, our approach uses training metrics to find a subset not necessarily representative of the whole, but rather one that can be emphasized within the whole dataset to improve performance, regardless of budget considerations. similarities can also be drawn between our approach and active learning @xcite @xcite , which searches unlabeled data for the most useful examples to label and uses them for training. @xcite aim to remove biases from models without re-sampling of the dataset, using product of experts between a weak (biased) model and a main model and achieving improvements over ood test-sets. karimi mahabadi et al. (2020) suggested a somewhat similar approach of de-biasing a main model by contrasting it with a "bias-only" model, to achieve ood improvements in tasks where the bias is known. @xcite also used biased models as a foil to a main model, to achieve de-biasing in vision-related tasks. utama et al. (2020) proposed a debiasing method that improves ood testing while maintaining id test results. their method regulates the model's confidence over biased examples in the dataset, using knowledge distillation in combination with biased models. this approach requires a-priori knowledge of the dataset's biases in order to formulate the biased model, and as such is not applicable to many nlp tasks. it also requires a large amount of compute, as it trains a full-size teacher model and a biased model besides the main model. ## conclusion we presented a new method for computing example training dynamics, which can be used to increase both id and ood performance, without any changes to model size or architecture. we demonstrated that etd can be transferable, i.e., they can be computed once and used many times for different models, reducing the computation cost at a long term. finally, we have shown that etd can be computed on the fly using dynamic training, which may hold the key to improved performance using etd at no extra compute cost. as the field of nlp leans more and more into the self-supervised pre-training paradigm, further research on etd may be focused on adjusting our method for larger and self-supervised datasets in order to improve and reduce the cost of pre-training as a whole.