Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model (2023)

Abstract: Pretrained language models have achieved remarkable success in various natural language processing tasks. However, pretraining has recently shifted toward larger models and larger data, which has resulted in significant computational and energy costs. In this paper, we propose Influence Subset Selection (ISS) for language model, which explicitly utilizes end-task knowledge to select a tiny subset of the pretraining corpus. Specifically, the ISS selects the samples that will provide the most positive influence on the performance of the end task. Furthermore, we design a gradient matching-based influence estimation method, which can drastically reduce the computation time of influence. With only 0.45% of the data and a three-orders-of-magnitude lower computational cost, ISS outperformed pretrained models (e.g., RoBERTa) on eight datasets covering four domains.

## introduction pretrained language models (ptms) @xcite @xcite , trained on massive and heterogeneous corpora, have significantly improved the state-of-the-art across a variety of natural language processing tasks @xcite . @xcite found power laws relating cross entropy loss to the sizes of language models and their training datasets. as a result, the field has recently shifted toward larger models and large data @xcite @xcite in hopes of improving performance. however, training a state-of-the-art language model requires substantial computational resources which demand considerable energy, along with the associated financial and environmental costs @xcite . for example, roberta-large @xcite , which was trained on 1000 v100 gpus for approximately one day, has a computational cost of 4.36×10 21 flops. recently, @xcite proposes palm, which consumes 580 times more flops than roberta-large. palm was trained on 6144 tpu v4 chips for more than 1200 hours, which is unaffordable for most researchers. therefore, finding ways to speed up pretraining is crucial for the development of pretrained model research. in general, there are three main strategies used to speed up pretraining in nlp: parallel architectures, efficient model architectures, and novel pretraining tasks. the first one is to train a single model utilizing multiple gpus distributed in many computational nodes @xcite @xcite . unfortunately, the gains in efficiency of this strategy depend entirely on the amount of computing hardware used. the second strategy is to improve model structures to reduce the computational complexity and therefore improve efficiency @xcite @xcite . the last one explores more challenging pretraining tasks to accelerate a model's convergence @xcite @xcite . however, their improvements are limited, with a reduction of less than an order of magnitude in computational expenses (measured in flops). in this paper, we aim to reduce the computational costs from data level (see motivated by influence function @xcite , we propose influential subset selection (iss) for language model, i.e. selecting the samples with the most positive influence on the downstream task. to calculate the label-aware influence value, iss utilizes the derivation chain rule from a test objective to training samples. nevertheless, directly applying the chain rule leads to computing the inverse of hessian with the complexity of o(nq 2 + q 3 )(n is the number of examples and q is parameter size), which is computationally expensive and may run out-of-memory in neural networks. to address this problem, we propose a gradient matching based influence approximation method for selecting pretraining data, which estimates the influence score by matching the gradient values of pretraining samples and end-task samples. our method avoids the computation of the inverse of hessian and significantly speeds up the estimation time of influence. our main contributions are summarized as follows: ## methodology we investigate an influence-based subset selection method to perform efficient pretraining while attempting to minimize accuracy loss on the end-task dataset (section 3.1). due to the high computational costs of influence function (koh and liang, 2017), we design an influence approximation strategy to speed up the calculation (section 3.2). ## experimental setup to evaluate the efficiency and generality of our approach, we conduct experiments in two settings: pretraining from scratch, and further pretraining. ## experimental results in this section, we will discuss the results of comparing our methods against other baselines. ## analysis 6.1 visualization of pretrained model we visualize the task data on iss-small, bert, and roberta, using the t-sne algorithm (van der @xcite . the results are shown in figure 4 . we can observe that the different classes of deep features in iss-small formed tighter clusters, suggesting that iss provides better initialization for downstream tasks. in contrast, the features learned by bert and roberta are distributed respectively in separate clusters with overlapping parts that could not be distinguished. ## conclusion in this paper, we propose influential subset selection for language model, which aims to reduce the computational costs of pretraining from data level. specifically, we introduce influence function to measure the importance of each pretraining sample. moreover, we design a simple, efficient, gradient matching-based method for influence estimation, which significantly speeds up the estimation time. experiments on various datasets demonstrate that our method achieves comparable performance with ptms, with a reduction of training flops by three orders of magnitude.