S peeding Up T ransformer Training By Using Dataset Subsampling - A n Exploratory Analysis (2021)

Abstract: Transformer-based models have greatly advanced the progress in the field of the natural language processing and while they achieve state-of-the-art results on a wide range of tasks, they are cumbersome in parameter size. Subsequently, even when pre-trained transformer models are used for fine-tuning on a given task, if the dataset is large, it may still not be feasible to fine-tune the model within a reasonable time. For this reason, we empirically test 8 subsampling methods for reducing the dataset size on text classification task and report the trade-off between metric score and training time. 7 out of 8 methods are simple methods, while the last one is CRAIG, a method for coreset construction for data-efficient model training. We obtain the best result with the CRAIG method, offering an average decrease of 0.03 points in f-score on test set while speeding up the training time on average by 63.93%, relative to the score and time obtained by using the full dataset. Lastly, we show the trade-off between speed and performance for all sampling methods on three different datasets.

## introduction in recent years, a great progress has been observed in the field of natural language processing thanks to use of deep neural network models and, most notably, transformer @xcite architecture-based models, such as bert @xcite . while transformer-based models achieve state-of-the-art results on a wide range of benchmarks @xcite , they are cumbersome in the number of parameters. consequently, computers with large amounts of ram memory and specialized hardware, such as multiple gpu or tpu devices, are required to train the models in reasonable time. as a result, such vast computational requirements put constraints on as one of the approaches to alleviate this problem, an extensive research has been conducted on methods for model compression, such as, model pruning, quantization, knowledge distillation, parameter sharing and others @xcite . although model compression methods can produce models characterising with low-latency and low-memory footprint for transfer learning, e.g. distilbert @xcite , but if the dataset is significantly large, it may still be not feasible to fine-tune the model within a reasonable time. therefore, if the model's size can't be further reduced without sacrificing its generalization performance, reducing the dataset is a task worth considering to further reduce training resource requirements. the challenge of reducing dataset size while trying to preserving model's performance is the main focus of this paper. previous research conducted on this topic mainly focuses on computer vision or traditional machine learning domain. pruning the dataset by removing noisy examples to improve model's performance was done in @xcite , while @xcite ranks examples by how valuable they are and takes the top ones in a greedy fashion w.r.t. the rankings. in instance selection methods, only the informative examples are selected prior to model training @xcite . similarly, a coreset (or weighted subset) construction was done in @xcite ; @xcite for specific traditional ml models while @xcite introduced a general approach applicable also to neural network models. in @xcite ; @xcite , the idea of knowledge distillation was used to distill a dataset to a small set of synthetic examples. to the best of our knowledge, there is no previous work on reducing the train set size for lowering the time and resource requirements for transformer-based model training. in this paper, we applied a general method for coreset construction described in @xcite to a transformer model. we empirically compared its performance against simpler subsampling approaches across 3 text classification datasets. the rest of the paper is organized as follows. in section 2, we describe methodology, including construction of the datasets, description of the sampling methods and experiment set up. in section 3, we show the performance across the datasets. and finally, in section 4, we discuss the results together with our outlook for future work. ## discussion as seen in figure 2 , on average and relative to the full dataset sampler, f-score and training time we further discuss the effect of conditioning on the maximum allowed training time ( finally, for the completeness of comparison, we tested the samplers on the same datasets but without preserving temporal information. only samplers which do not require temporal information to work were taken into account. again, we obtained top results by using craig sampler (appendix