Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values (2023)

Abstract: Although Shapley values have been shown to be highly effective for identifying harmful training instances, dataset size and model complexity constraints limit the ability to apply Shapley-based data valuation to fine-tuning large pre-trained language models. To address this, we propose TS-DShapley, an algorithm that reduces computational cost of Shapley-based data valuation through: 1) an efficient sampling-based method that aggregates Shapley values computed from subsets for valuation of the entire training set, and 2) a value transfer method that leverages value information extracted from a simple classifier trained using representations from the target language model. Our experiments applying TS-DShapley to select data for fine-tuning BERT-based language models on benchmark natural language understanding (NLU) datasets show that TS-DShapley outperforms existing data selection methods. Further, TS-DShapley can filter fine-tuning data to increase language model performance compared to training with the full fine-tuning dataset.

## introduction large language models (lms) have achieved stateof-the-art performance on many natural language processing (nlp) tasks @xcite @xcite . to adapt these models to new datasets and tasks, the standard approach is to fine-tune a pre-trained lm on a targeted downstream task. this allows the pre-trained general linguistic knowledge to be leveraged while fine-tuning to learn the task-specific information. however, during fine-tuning, pre-trained lms are prone to significant performance degradation in the presence of noisy data @xcite . this effect may be further amplified when noisy or otherwise harmful instances are highly influential to the model parameters @xcite . as a result, it is important to identify harmful in-figure 1 : an overview of ts-dshapley: 1) process the data using the target lm; 2) compute sampling chains using a subset of the training set and aggregate the resulting shapley values; and 3) transfer the estimated data value information for use with the target lm by estimating the optimal low value data removal index. to automatically identify harmful data, prior works have used training dynamics @xcite and estimation of marginal contributions via leave-one-out retraining @xcite or influence functions @xcite . shapley values, which satisfy certain desirable fairness guarantees, have also recently been adopted from cooperative game theory to measure datum contributions, where a data point's shapley value is the average marginal contribution to every possible data subset @xcite . in practice, shapley-based data values are approximated using various techniques @xcite @xcite @xcite , as exact shapley value computation over a dataset would require exhaustively retraining the model for every datum on every possible subset (i.e. exponential complexity with respect to the number of data points). however, many of the existing approximation methods still exhibit a computational bottleneck when considering datasets and models at scale (e.g. datasets larger than 5k instances). this, in turn, directly limits the application of shapley-based data valuation to state-of-the-art lms and many nlp datasets. to address the challenges posed by 1) the model constraint (the model retraining requirement) and 2) the dataset constraint (the time-complexity/dataset size relation), we propose transferred sampling data shapley (ts-dshapley), an algorithm that utilizes two novel components that directly address each constraint. specifically, to address the model constraint, we propose to compute shapley-based data values using a simple, linear model that is trained on the learned representation from the target lm. additionally, to address the dataset constraint, we propose a sampling-based method that computes shapley values on data subsets and aggregates them for valuation of the entire training set. our contributions are as follows: 1) we propose a sampling-based data shapley computation method and demonstrate its efficacy empirically using as little as 2% of the original training data; 2) we propose the use of a simple linear classifier with a target model's pre-trained representation and demonstrate empirically the performance gains achieved over alternate pre-trained embeddings; and 3) we show the efficacy of shapley-based data valuation and selection methods on benchmark nlu tasks using fine-tuned large lms. foot_0 2 related work while shapley values are often applied in a post hoc manner following model training @xcite @xcite @xcite , the demonstrated efficacy makes it a natural extension to apply such methods for data selection prior to training. to this end, shapley values have been used for evaluating data for transfer learning (parvez and chang, 2021) and in active learning @xcite . further, although shapley-based data values have primarily been considered model-specific, in practice, a subset of training instances that may harm performance may be mislabeled @xcite @xcite or exhibit spelling mistakes or grammatical errors @xcite , which should be intrinsic to the dataset. prior works have demonstrated the transferability of shapley-based data values across various classifier architectures @xcite and have demonstrated the efficacy of surrogate knn classifiers using pre-trained embeddings @xcite . notably, our work differs in that we utilize the pre-trained embeddings extracted from the target lm and avoid the k-nearest neighbor assumption that training data far from a test datum do not contribute to its prediction @xcite . for each training instance i, the shapley value ϕ i is defined as the average marginal contribution of i to every possible subset s ⊆ d that contains this instance @xcite : ## conclusion in this work, we propose ts-dshapley to address the model and dataset constraints that currently contribute to a computational bottleneck when computing shapley-based data value estimates.