DEFT - UCS : Data Efficient Fine-Tuning for Pre-Trained Language Models via Unsupervised Core-Set Selection for Text-Editing (2024)

Abstract: Recent advances have led to the availability of many pre-trained language models (PLMs); however, a question that remains is how much data is truly needed to fine-tune PLMs for downstream tasks? In this work, we introduce DEFT-UCS, a data-efficient fine-tuning framework that leverages unsupervised core-set selection to identify a smaller, representative dataset to fine-tune PLMs for text-generation needed for text editing tasks such as simplification, grammar correction, clarity, etc. We examine the efficacy of DEFT-UCS across multiple text-editing tasks, and compare to the state-of-the art text-editing model, CoEDIT. Our results demonstrate that DEFT-UCS models are just as accurate as CoEDIT, across eight different datasets consisting of six different editing tasks, while finetuned on 70% less data.

## introduction how much data do we need to fine-tune a pretrained language model (plm) for a specific downstream task? while successes in language modelling have led to numerous publicly available plms and ability to produce fine-tuned models for downstream tasks -the answer mostly remains, "as large as possible, and of good quality". for example, alpaca, an instruction-following model, is trained with 52k data samples @xcite . similarly, copoet, a collaborative poetry writing system is fine-tuned using 87k data samples @xcite . metamath, a mathreasoning llm is fine-tuned with 395k data samples @xcite . although fine-tuning plms on specific task results in performance gain, acquiring large amounts of data for fine-tuning is not easy for real-world applications which often require niche knowledge and domain expertise. researchers have explored variety of methods primarily focused on improving the computational * contributed during an internship at accenture labs, sf efficiency of fine-tuning, including parameterefficient fine-tuning approaches (peft) to reduce computational costs by optimizing parameter updates @xcite as well as leveraging active-learning for iteratively selecting data samples during training @xcite . instead, our work focuses on improving the data efficiency of plm fine-tuning without requiring iterative fine-tuning. specifically, we explore how to fine-tune plms with significantly less data samples and without a cost to model performance. related to language models, researchers have experimented with different core-set selection metrics @xcite to improve the data efficiency during pre-training. @xcite demonstrated how perplexity, l2-error norm (el2n) and memorization can be utilized to select smaller, good quality datasets for pre-training. similarly, @xcite leverage el2n to dynamically remove data samples with high el2n between training epochs. however, these metrics assume access to task data and reference models to perform dataset pruning. in real world applications, utilizing such supervised, data-pruning metrics are less realistic since large amounts of annotated task-specific data may be costly to acquire. this leads us to our main research question: how can we leverage unsupervised data pruning to fine-tune plms for downstream tasks in a more data efficient manner? in this work, we introduce a new data-efficient fine-tuning framework, deft-ucs, that uses unsupervised core-set selection to minimize the amount of labelled data needed to fine-tune plms for text generation related to multiple text-editing tasks. our framework is inspired by @xcite , who utilize clustering-based dataset pruning to reduce training samples for image-classification models, and to the best of our knowledge, our framework is the first to leverage unsupervised core-set selection for data-efficient fine-tuning of plms. we study the utility of deft-ucs in fine-tuning plms for text-generation across eight different datasets consisting of six different text-editing tasks, and compare deft-ucs models to the stateof-the-art text-editing model, @xcite . our contributions are as follows: ## related works efficient fine-tuning of llms most work on efficient fine-tuning techniques for llms have primarily focused on parameter-efficient fine-tuning (peft) approaches @xcite , improving computation efficiency by updating a subset of model parameters. recently, there has been an increasing focus on improving the data-efficiency of llms, considering how to pre-train and fine-tune llms with smaller subsets of data @xcite @xcite @xcite . for instance, @xcite introduce lima, an approach to fine-tune llama @xcite with only 1k diverse and high quality samples. however, the lima approach is underspecificed without a general subsampling procedure. also, chen et al. (2023) develop skill-it!, which creates efficient datasets by learning hierarchical relationships between samples. however, identifying hierarchical relationships is non-trivial and not all datasets may include them. more closely related to our work, ivison et al. (2022) leverage k-nearest neighbors to learn multiple data-efficient fine-tuned models for individual tasks. instead, we aim to learn a single data-efficient fine-tuned model that performs competitively across a variety of datasets. similarly, marion et al. (2023) utilize perplexity and el2n, to find smaller datasets for llm pre-training, and attendu and corbeil (2023) uses el2n to iteratively remove unimportant samples during fine-tuning. both marion et al. (2023) and attendu and corbeil (2023) assume access to task data to train various reference models for few epochs to calculate el2n and perplexity. in contrast, we leverage unsupervised core-set selection, omitting the need for any reference model during the dataset sampling step. core-set selection & dataset distillation several works in ml have developed variety of coreset selection @xcite and dataset pruning @xcite methods to find smaller subsets of data needed to train deep learning models without model performance loss. craig @xcite finds coresets by approximating gradient calculations, while retrieve (killamsetty et al., 2021) finds coresets by optimizing for model loss. also, yang et al. (2022) utilize influence functions (koh and liang, 2017) to prune redundant samples. a unifying idea among these methods is the need for labelled data. alternatively, core-set selection methods for unlabelled data have used clustering methods. birodkar et al. (2019) use agglomerative clustering to find semantic similarities among data points and prune redundant samples. similarly, sorscher et al. ( 2022 ) use vanilla k-means clustering and distances to cluster centroids for pruning easy and hard samples. recently, data distillation algorithms have also been developed to improve data-efficient model training @xcite . typically, data distillation methods generate new synthetic datasets in which data samples are edited to preserve more information for performance generalization @xcite . our work considers efficient core-set selection without the generation of a synthetic dataset. specifically, our work extends @xcite , which performs easy and hard sampling to reduce training data from a single dataset, imagenet @xcite . in our work, we additionally consider the effects of random sampling (mix of easy and hard), access to initial seed data, and study the effects of such sampling techniques for dataset pruning across several text-editing tasks. instruction tuning for text-editing recently, instruction tuning of plms has shown impressive success in its ability to enable plms to follow instructions as well as improvement in generalization across various tasks in zero/few shot settings @xcite . training models to explicitly follow natural language instructions has become increasingly popular for text-editing tasks as well. @xcite develop rewritelm by fine-tuning palm @xcite variants for the task of rewriting long-form texts. similarly, @xcite develop peer by finetuning t5 @xcite variants to emulate the collaborative writing process. additionally, raheja et al. (2023) develop coedit by fine-tuning flan t5 (chung et al., 2022) models to perform single and compositional edits across multiple edit tasks. furthermore, zhang et al. (2023) produce an instruction-tuned llama model that improve textediting capabilities. a commonality across these works include the usage of large scale datasets for fine-tuning. for example, coedit (raheja et al., 2023) and zhang et al. (2023) leverage datasets with 82k and 60k examples, respectively. in our work, deft-ucs maximizes model performance of fine-tuned models in a data efficient manner by finding a representative, smaller dataset needed for fine-tuning. we investigate the efficacy of our deft-ucs framework to instruction fine-tune plms for eight text-editing tasks. ## problem formulation we formulate deft-ucs as an unsupervised coreset selection problem @xcite in contrast to existing dataset pruning methods which primarily use supervised core-set selection @xcite . specifically, let d represent an existing large dataset, p represent a plm, and m d represent p fine-tuned on d. our deft-ucs framework aims to find a representative core-set d c ⊂ d such that leveraging d c can fine-tune p and result in a finetuned model m dc with comparable performance to m d . note, we refer to comparable evaluation performance in the form of both quantitative nlp metrics and qualitative human evaluations. specific to unsupervised core-set selection, deft-ucs finds d c without needing d to include annotations or labels. thus, we find d c by only using the input samples {x 1 ..x n } within d. these input samples, in the context of instruction fine-tuning, represent task instructions and input texts. to perform unsupervised core-set selection, we build upon the sota clustering-based core-set selection method by sorscher et al. (2022), given its extensive evaluations against other supervisedbased core-set selection methods. while sorscher et al. (2022) demonstrate the efficacy of clusteringbased core-set selection for imagenet (deng et al., 2009), our work is the first to investigate the effectiveness of clustering-based core-set selection in non-classification tasks, such as fine-tuning plms for multiple text-editing tasks. algorithm 1 unsupervised core-set selection (ucs) input: d @xmath0, x 1 ...x n } -large dataset input: k -num. of clusters input: a -amount of samples per cluster input: α, β, -sampling weights output: d @xmath1..x p } -core-set 1: d @xmath2: d @xmath3: cl 1:k , ce 1:@xmath4, k) 4: for i in k do 5: for d in cl i do 6: dist @xmath5, ce i ) 7: end for 8: dist @xmath6: d @xmath7: α*a] + dist sorted [ -β*a:] 10: d @xmath8, d c ) 11: end for 12: return d c 4 deft-ucs framework we consider a scenario in which there exists an initial amount of data, d base ⊂ d, that is sampled in a stratified manner to provide an overall representation of the downstream fine-tuning task. let d remain represent the remaining data after d base is sampled. the goal of ucs is to then find a core-set d c ⊂ d remain that enriches d base such that d c and d base , together, form a representative subset that can be used to fine-tune a plm and result in a fine-tuned model m def t -u cs with comparable performance to m d , a plm fine-tuned with d. in algorithm 1, we detail the crux of our deft-ucs framework, the ucs method. ## deft-ucs applied to text-editing we evaluate the utility of deft-ucs in the context of instruction-based fine-tuning for multiple text editing tasks. to our knowledge, the current sota instruction fine-tuned text-editing lm is coedit (m coedit ) foot_0 trained on dataset d coedit @xcite . overall, d coedit includes 82k good-quality edit instructions spanning six different edit-tasks (raheja et al., 2023) (d coedit detailed in appendix a.1). given the data quality in d coedit and sota performance of m coedit , we apply deft-ucs to d coedit . below, we detail the hyper-parameter choices in deft-ucs in the context of d coedit . ## results our results below show that deft-ucs can provide a data-efficient method for producing competitive fine-tuned models for six different text-editing tasks. ## conclusion we introduce deft-ucs, a data-efficient finetuning framework that leverages unsupervised coreset selection to find the minimum amount of data needed to fine-tune a plm for text-editing tasks. our best performing deft-ucs model, fine-tuned with only 32.5% of the coedit dataset @xcite , has comparable performance to the sota coedit @xcite on two text-editing tasks, and improved performance to @xcite on four text-editing tasks, and the lima approach @xcite . human evaluators also preferred edits generated by deff-ucs model over coedit. these results show the overall utility of our deft-ucs framework towards data-efficient finetuning of pmls in text-editing tasks. to better understand the generalizability of deft-ucs, we plan to first apply it to more text-generation tasks in the future. subsequently, we aim to benchmark the efficacy of different data-sampling strategies across various plms for these tasks. limitations first, the hyper-parameters within the ucs algorithm of our deft-ucs framework are selected manually using task specific knowledge. future work should consider how to automate the selection of these hyper-parameters. additionally, while our ucs algorithm within deft-ucs uses the distance between data samples and centroid distance to define sampling methods, future work should explore other sampling methods informative to nlp tasks. additionally, we show the benefit of deft-ucs in the context of text generation across eight text-editing datasets. however, future work should benchmark deft-ucs across other diverse nlp tasks, beyond text generation, such as summarization or text expansion. moreover, while our human evaluation shows that deft-ucs produced models generate accurately edited texts, future work should conduct larger-scale user studies to understand human perceptions of across multiple qualtitative dimensions. more work is also required to explore the benefit of deft-ucs in fine-tuning other plms for downstream nlp tasks, and comparing the benefits of deft-ucs with peft @xcite approaches, and whether deft-ucs with peft can further improve the fine-tuning efficiency of plms.