Mitigating the Burden of Redundant Datasets via Batch-Wise Unique Samples and Frequency-Aware Losses (2023)

Abstract: Datasets used to train deep learning models in industrial settings often exhibit skewed distributions with some samples repeated a large number of times. This paper presents a simple yet effective solution to reduce the increased burden of repeated computation on redundant datasets. Our approach eliminates duplicates at the batch level, without altering the data distribution observed by the model, making it model-agnostic and easy to implement as a plug-and-play module. We also provide a mathematical expression to estimate the reduction in training time that our approach provides. Through empirical evidence, we show that our approach significantly reduces training times on various models across datasets with varying redundancy factors, without impacting their performance on the Named Entity Recognition task, both on publicly available datasets and in real industrial settings. In the latter, the approach speeds training by up to 87%, and by 46% on average, with a drop in model performance of 0.2% relative at worst. We finally release a modular and reusable codebase to further advance research in this area.

## introduction deep neural networks have recently enabled impressive results across many fundamental tasks @xcite . however, training state-of-the-art models is now a very demanding process, in terms of both time and resources @xcite . the issue is exacerbated when models are required to train on datasets that naturally exhibit a redundant distribution. user queries are one such example: aol @xcite and msn query logs @xcite are composed for the 51.6% and 52.4% of duplicates, respectively. a similar phenomenon can be observed in recommender systems data, in which most of the entries involve a relatively small set of popular items @xcite . finally, in large scale conversational assistant data, the vast majority of user interactions is composed of commands and queries that are frequently expressed with minimal or no variation. when training on this redundant data, there will be instances in which: (i) the training input is the same, and (ii) the model has the same weights; in these instances repeated computations will occur, increasing training times. in this paper, we propose a simple yet effective approach to reduce repeated computations during training by removing duplicates at the batch level, while accounting for frequency of the samples in the batch during the loss computation. this leaves the data distribution perceived by the model untouched and we empirically show that it leads to a model that has similar weights and has followed a similar trajectory in the parameter space during training, compared to a model trained without any data deduplication. our contribution is therefore four-fold: (i) we propose a novel online deduplication technique to mitigate the training burden over redundant datasets and provide rigorous mathematical reasoning backing its benefits; (ii) we show its effectiveness on both a real industrial datasets as well as artificially upsampled public datasets; (iii) we further provide a set of empirical analyses, including a study of the effect on the parameters' evolution and the difference in benefit when varying batch size; (iv) finally, we release a modular and extensible codebase 1 , implementing deduplicator classes, easily reusable by the community. even though the methodology is agnostic to both task and model, we experimentally validate its strengths on ner as it constitutes a critical task for large-scale conversational assistants, on which we show the duplication problem to be relevant. we believe this work fills a gap in the current research landscape, since deduplication techniques for training data appear to be an under-explored research territory, but also an increasingly pressing need in industry. ## related work the success of language models pre-trained on large corpora, such as bert @xcite , raised awareness on optimization of training times and costs within the nlp community. @xcite showed that carbon footprint of nlp research is following a concerning trend and spurred researchers to prioritize the development of computationally efficient algorithms. countless directions have been explored by the community, from distillation techniques @xcite ) used to produce smaller models @xcite ) up to efficient computation frameworks to improve distributed training @xcite ). to the best of our knowledge, only a few works in the literature address the problem of optimising training in presence of duplicates in the data. @xcite show the benefits of completely removing duplicates when pre-training large language models. @xcite propose a way to improve accuracy focusing on mini-batches and performing undersampling and oversampling in order to balance classes. @xcite mention the possibility of reducing computation time by removing all but one of the duplicates in a mini-batch, 1 available at github.com/amazon-science/unique-batches. although their work focuses on the optimal way to sample data to minimize gradient variance, and not on training time reduction. similarly, @xcite propose a way to balance the training effort among batches, to improve stochastic gradient descent (sgd) by minimizing gradient variance. compared to these previous works, our approach focuses on reducing the repeated computation that occurs when training on redundant datasets, retaining model performance while being minimally invasive to the pre-existing setup. ## approach a deep learning model m(θ) is typically trained over a given dataset d with sgd (or one of its variants), estimating the gradient of a loss function with respect to the model weights θ, by iterating over the dataset in batches of fixed size. if d contains duplicates, then some of them might fall within the same batch, resulting in the same computations occurring multiple times. we propose to remove duplicates while building the batch, also accounting for the number of occurrences of the samples when computing the loss. in practice, first creating the batches and then removing the duplicates would result in smaller batch sizes and therefore in under-exploiting the parallel computation enabled by gpus. therefore, we keep the actual batch size the same, filling a batch with unique samples (ignoring repetitions when encountered, see fig. 1 ), but accounting for repetitions by multiplying the loss of a sample by its frequency in the batch. see appendix a.1 for the details of the procedure. by considering the number of repetitions of the samples, the size of the batches remains the same while virtually containing more samples, therefore reducing the required number of batches to iterate over the dataset and leading to training time reduction. we remark that the proposed technique does not make any assumption on the underlying task and model, but only affects the way data is loaded during training. therefore, it is well suited for a production setting in which one wishes to reduce the burden of frequent re-training, while at the same time changing the setup as little as possible. motivation our methodology is grounded on the following intuitions: (i) frequent samples should be given a larger weight than rarer ones, as they are most frequently encountered in the production traffic, and (ii) the model should be able to see the duplicated samples multiple times per epoch, i.e. at different stages of the model parameters evolution. while (i) explains why we should deduplicate and take sample frequency into account in the loss computation, (ii) explains the need to do it at the batch level. in section 6 we provide empirical support to these observations. ## boost estimation estimating the expected increase in virtual batch size is important for two reasons. first, the virtual batch size allows us to compute the expected reduction in the number of batches required to iterate over the dataset. this in turn provides an estimate for the reduction in training time (see eq. ( 3 )) without the need to actually run any training. second, it is common practice to scale the learning rate when increasing the batch size @xcite , thus having an estimate of the virtual batch size helps correcting the learning rate. this is discussed more in detail in section 7. to the best of our knowledge there is no closed form solution for e b virtual in eq. ( 2 ). therefore we derive a solution for the number of duplicates d in a batch b of size n, and then invert the formula to compute the expected number of unique samples in b as @xmath0. this way a numerical solution to the original problem can be found by iterating over the possible values of @xmath1, . . . , n , up to the one that yields @xmath2. in the following we introduce the formula for d and leave the details of its derivation in appendix a.2. consider a dataset of n samples, some of which may be duplicates, taken from a collection then, we can show that: effect of sampling strategy the estimate in eq. ( 8 ) assumes batches are formed by sampling uniformly at random from the dataset. however, nlp practitioners often rely on sampling strategies that optimize memory consumption, such as bucketing by sequence length @xcite . samples are distributed in buckets based on their length, and then batches are formed sampling uniformly at random from these. in such a scenario, the boost estimation in eq. ( 8 ) still holds, but on each bucket individually. the overall expected number of duplicates can be computed as a weighted average of the estimates on each bucket, weighted by the bucket size. we highlight that if samples are grouped in buckets by length, then all of the duplicates of one sample will fall in the same bucket. this has the effect of increasing the number of expected duplicates in each batch (n in eq. ( 8 ) is smaller), leading to larger training time reduction in realistic scenarios that use bucketing. ## experimental setting the proposed approach is tested on the task of ner @xcite . two samples (also referred to as utterances) are considered duplicates if they share the same annotation, i.e. word-level tokens and corresponding ground truth ner labels in the dataset. the experiments are performed on datasets used for training a large-scale conversational assistant (referred to as internal in the following) and also on publicly available data. the internal datasets comprise live traffic utterances, de-identified for privacy regulations and annotated to enable supervised training of deep learning models for solving the ner task. the datasets exhibit varying degree of skewness in their redundancy, and we refer to them as internal ms , internal vs , internal xs , meaning mildly skewed, very skewed and extremely skewed, respectively. given the artificial deduplication of manually curated datasets, we upsample a public dataset to mimic the redundancy observed on internal data. the mitrestaurant dataset @xcite , consisting of restaurant-related queries, is chosen for the semantic similarity of its queries to the utterances found in the internal datasets. from this dataset, upsampling is performed to arrive at three datasets with redundancy ratios of r 1 = 0.5, r 2 = 0.7 and r 3 = 0.9. we refer to these artificially upsampled public datasets as mit ms , mit vs , mit xs , with the same meaning of the acronyms. given the redundancy ratio r of interest for each of the three datasets, these are generated as follows. first, the number of duplicates to draw is computed, according to the above probability distribution (eq. ( 8 )), as @xmath3. then, to generate a realistic distribution, the fact that shorter utterances are more frequent @xcite and thus more likely to be duplicated is leveraged. in a conversational assistant, for instance, we expect to observe more frequently short utterances such as "yes" or "stop" than long sentences related to some more specific queries. this heuristic is implemented by sampling the d duplicates according to a power-law over the length of the utterance in characters, for which an utterance u i with length l i is sampled with probability p(u i ) = 1 (l i ) α . the exponent of the power-law α is set to 3 for mit ms and mit vs , while it is set to 5 for mit xs . see we compare the proposed approach with various baselines (see all the deduplication methods are applied to the training of a model with the same architecture, across all experiments. in particular, we employ a simple ner architecture that obtains contextual word embeddings from a pre-trained distilbert encoder @xcite . the embeddings are then fed to a multi-layer perceptron (mlp) to map into the label space. the models are trained to convergence with early stopping. we remark that what changes across the experiments is the deduplicator and not the model that is employed. the latter is in fact defined by the same architecture and hyperparameters, except for the learning rate that is adjusted as described in section 7 to account for the difference in virtual batch size. refer to appendix a.4 for further details. ## experimental results results on internal data we observe that introducing the sample weighting term in the loss when deduplicating batch-wise (bwu) does not seem to result in a significant improvement over the un-weighted variant (bu). we hypothesize this to be a consequence of leaving the pre-trained bert encoder frozen during training. to test this hypothesis, we experiment also on a simpler lstm-based model @xcite , without pre-training (see results on public data ## discussion influence of batch size as can be seen in figure 2 , the expected number of duplicates increases with the batch size. intuitively, given a finite set of elements that contains repetitions, the larger a sample, the more likely it is to have repetitions. the plot clearly shows how the greatest increase in the number of duplicates is obtained when passing from small batch sizes to over 1000. the maximum possible reduction in number of batches would trivially be obtained when the batch size is equal to the total number of samples, when the reduction would be exactly the redundancy ratio of the dataset. learning rate it is advisable to scale the learning rate when increasing the batch size, as the gradient computed over a larger batch size is a more accurate estimate of the real gradient and should therefore provide a more reliable direction. small batches naturally provide more chaotic gradients and therefore a large learning rate may result in "overshooting" and missing the minima. a common approach is to increase the learning rate as the square root of the batch size increase factor @xcite , or to scale it linearly with the batch size @xcite . we adopt the latter in this work, multiplying the learning rate by the expected increase in batch size computed as in section 3. ## conclusions in this paper, we address the problem of reducing training time when using redundant datasets, proposing a novel approach agnostic to task and model that results in a significant time reduction without waiving performance. the approach is compared with various deduplication methods on the task of named entity recognition, on both industrial and public datasets. the comparison shows that our approach is the only one to significantly reduce training time while maintaining the same model performance metrics, with observed boosts in training time of 23%, 47% and 87% on datasets used to train models for a large-scale conversational assistant. various analysis are conducted, on the tradeoff with batch size and on how learning trajectories are modified. we also provide a theoretically sound procedure to estimate the expected reduction, allowing practitioners to assess the benefits before employing the method. finally, a modular and reusable codebase is released to foster further research in the area. we believe that this approach can have a high impact on the industry, where large, expensive models are often trained on datasets containing redundant user queries or items. this reduction in training times may in fact allow for faster experimental iterations while cutting times and costs, also reducing the carbon footprint of deep learning models. as future work we plan to (i) leverage the generality of the approach on other tasks that exhibit high redundancy in data, like semantic search and (ii) study a more relaxed definition of equality between two samples, to allow considering two samples the same if they only differ up to a tolerated quantity.