I n2 C ore: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Models (2024)

Abstract: Despite advancements, fine-tuning Large Language Models (LLMs) remains costly due to the extensive parameter count and substantial data requirements for model generalization. Accessibility to computing resources remains a barrier for the open-source community. To address this challenge, we propose the In2Core algorithm, which selects a coreset by analyzing the correlation between training and evaluation samples with a trained model. Notably, we assess the model’s internal gradients to estimate this relationship, aiming to rank the contribution of each training point. To enhance efficiency, we propose an optimization to compute influence functions with a reduced number of layers while achieving similar accuracy. By applying our algorithm to instruction fine-tuning data of LLMs, we can achieve similar performance with just 50% of the training data. Meantime, using influence functions to analyze model coverage to certain testing samples could provide a reliable and interpretable signal on the training set’s coverage of those test points.

## introduction with the advent of large language models (llms) exhibiting surprising abilities across a variety of language tasks @xcite @xcite @xcite @xcite , open-source language models have surged in popularity as part of broader efforts to democratize their accessibility @xcite . furthermore, performance can be improved with custom data that is smaller than their pretraining data, which is known as supervised fine-tuning @xcite . one type of fine-tuning important to the open-source community is instruction tuning, which allows models to follow a broad or specific-set of instructions and to discuss with users in natural dialogue @xcite . however, instruction-tuning of open-source llms remains limited in multiple areas. one prominent limitation is the expensive cost of fine-tuning such models, caused by a large number of parameters and data required @xcite . the popular approach is to scale-up the parameter count and the dataset size, increasing the computation required to train the models. this method to improve llms does not lend-well to organizations without massive computing resources, such as the majority of entities who rely on opensource. another limitation is evaluation, especially for instruction-following models. evaluation is challenging as a result of llms' inherent openended generation, where a space of ideal outputs exists, and existing evaluations usually measure ability in a specific area, such as summarization. furthermore, evaluation sets may be hard to construct for domain-specific abilities. these two issues limit the continued adoption of open-source models. to alleviate the cost of fine-tuning, many have focused on improving the hardware @xcite or the algorithm of the training process @xcite . to improve evaluation, many have introduced more sophisticated benchmarks @xcite or rely on automated evaluation using more powerful llms @xcite . however, one prominent limitation faced by these approaches is that they do not analyze how the training data can be manipulated to induce a better-performing trained model. by focusing on the problem of coreset selection, we instead take a data-centric approach using influence functions. influence functions are a method from robust statistics @xcite and are first adapted to neural-network-based machine learning models by @xcite . they approximate how much the model's prediction changes when a particular training point is removed from the training process via the model gradients. in this work, we aim to show that influence func-tions are versatile statistical tools that can address two questions on data suitability: 1) what training points are suitable for the test set? and 2) what test points are suitable given that the model was already trained on a particular training set? we use influence functions to identify influential data to address the two aforementioned issues by making fine-tuning more efficient by using less data and identifying test points well-covered by the finetuned model. our method is orthogonal to concurrent efforts to make fine-tuning more efficient for llms that focus on modifying the model architecture, such as simplifying transformer blocks (he and hofmann, 2023), or that focus on reducing the number of trainable parameters, such as lora @xcite . it can therefore be combined with these methods. in particular, our method relies on datainf @xcite , which works on top of lora to calculate influence values. our main contributions can be summarized as follows: ## influence functions for coreset selection in this section, we explain what influence functions are, their use in assigning a value for each training datapoint in in2core, and an algorithmic improvement to use less memory for the influence function algorithm we use, namely datainf @xcite . we then explain the in2core algorithm and perform experiments to showcase the performance of different subset selections of the data. finally, we discuss the implications of having a smaller and better-quality training dataset, namely faster training and comparable performance. ## in2core for model coverage in this section, we use in2core at test-time. specifically, we use influence values to measure how "suitable" individual test points are to a final model trained on a given training set. we first describe the rationale and our method for analyzing model coverage. then, we perform an experiment comparing our method to analyzing model coverage via semantic similarity. finally, we discuss the implications of the experiment. note that we use the gemma-2b model fine-tuned on the proponents dataset for our experiments, described in 3.4, and thus omit an experimental setup section. evaluating a trained instruction-tuned model is difficult considering the space of acceptable outputs given certain questions. rather than solely looking at discrete outputs to determine if the model generalizes to a test point, we instead augment this analysis by also looking at the training set. we capture the influence of the entire training set by taking its average gradient, thereby giving a single influence value for each test point. this is akin to measuring how much a test point falls within the training distribution. to the best of our knowledge, this is the first time that influence values have been used in the literature to analyze how a given training set as a whole is appropriate for a given test point via a trained model. importantly, this method is compatible with different evaluation metrics. in section 4.1, we specifically apply this on the perplexity metric. for instruction-tuned llms, having an indication on whether a given test input fits within its training distribution is essential because there is either an absence of a ground-truth to validate with or a ground-truth that is hard to articulate (e.g. a writing task where there is a space of acceptable outputs). we would like to emphasize that influence functions should not be used by themselves to evaluate a model's capability. it may be the case that the entire training set is beneficial to the model at learning a specific capability, but the training set size is insufficient for the model to learn that capability. rather, influence functions provide further guidance at understanding model capability at evaluation by tying that capability to the training set, especially when one wishes to determine how to improve the model (as in coreset selection). our method can be integrated in pipelines for model debugging (which test points does the model need more data for?) and handling inference requests (which requests is the model not suited to perform well given its training?) in productiongrade llms. furthermore, using influence functions in this manner can be computationally-efficient. we can exploit a feature of influence functions to our advantage: the most expensive computational step is calculating the gradients, but once the gradient computation is done, we can store them. in particular, we store the average gradient of the training dataset, and we can arbitrarily use it for future inference on unseen test points. while we still need to compute the model gradients with respect to new test points, the largest cost, coming from the training set, is only performed once. ## conclusion in this work, we have shown that blindly adding data into the training set can degrade the model's performance. we developed in2core, an algorithm for coreset selection based on influence functions. our algorithm is practical and can be combined with other pre-processing steps to make supervised fine-tuning more efficient. we show that training on just half of the original training data can be comparable with the performance of a model trained on the full data for a variety of reference and target models. importantly, (1) we observe a transfer effect where a reference model of a different architecture can be used to select the coreset data, and (2) this effect holds even with a reference model that is smaller than the model we are fine-tuning. finally, we explore the use of influence to measure a trained model's coverage. we show that measuring data influence and an evaluation metric can aid in efficient model training than merely using an evaluation metric alone. we also show that this "influence" computed via influence functions captures a model's coverage better than by semantic similarity.