Influence Scores at Scale for Efficient Language Data Sampling (2023)

Abstract: Modern ML systems ingest data aggregated from diverse sources, such as synthetic, human-annotated, and live customer traffic. Understanding which examples are important to the performance of a learning algorithm is crucial for efficient model training. Recently, a growing body of literature has given rise to various “influence scores,” which use training artifacts such as model confidence or checkpointed gradients to identify important subsets of data. However, these methods have primarily been developed in computer vision settings, and it remains unclear how well they generalize to language-based tasks using pretrained models. In this paper, we explore the applicability of influence scores in language classification tasks. We evaluate a diverse subset of these scores on the SNLI dataset by quantifying accuracy changes in response to pruning training data through random and influence-score-based sampling. We then stress-test one of the scores – “variance of gradients” (VoG) from Agarwal and Hooker (2022) – in an NLU model stack that was exposed to dynamic user speech patterns in a voice assistant type of setting. Our experiments demonstrate that in many cases, encoder-based language models can be fine-tuned on roughly 50% of the original data without degradation in performance metrics. Along the way, we summarize lessons learned from applying out-of-the-box implementations of influence scores, quantify the effects of noisy and class-imbalanced data, and offer recommendations on score-based sampling for better accuracy and training efficiency.

## introduction a salient challenge in training transformer-based models is selecting which examples are most important for learning. understanding the relative importance of training examples towards model performance can inform data selection strategies that minimize customer privacy risks associated with the collection of training data, estimate the impact of the removal of copyrighted or sensitive data, determine mixing strategies to augment monolingual and multilingual datasets to improve accuracy, and identify defective subsets of data. at the same time, in cases where it is desirable to train on as much data as possible -such as large language models -determining the influence of different data instances (both contextually and during pretraining) can help identify failure modes at the level of specific tokens @xcite , determine the impact of removal of intellectual property, and significantly reduce costs through more efficient model training @xcite . a growing body of literature in the science of deep learning aims to capture this hierarchy of example importance and has led to a proliferation of a number of "difficulty" or "influence" scores (e.g., @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; see app. a for a more complete review). these scores use various training artifacts, such as the margin of confidence or the variance of loss gradients, to rank the relative contribution of each example to model performance. this ranking of examples can then be used in many downstream tasks that require intelligent data selection, such as pruning datasets while maintaining or even improving model accuracy @xcite @xcite ; identifying outliers and misannotations in labeled data @xcite @xcite @xcite ; or reweighting/reordering training examples to increase model robustness @xcite . apart from a few notable exceptions @xcite @xcite , influence scores have primarily been developed and demonstrated in the context of image classification, and relatively little is known about their efficacy in downstream language-based tasks. 1 the application of these scores to data selection is further complicated by the fact that during fine-tuning, modern ml systems often ingest a vast amount of data that come from multiple sources, such as synthetic, weak signal, 2 live customer data, and humanannotated. beyond quantifying the efficacy of influence scores in this highly mixed data setting, there is an operational question of the existence of a simple, scalable influence score that can be easily accommodated in a production workflow. in this work, we take a first pass at answering these questions. first, we benchmark a subset of influence scores on the snli dataset @xcite in the downstream task of data reduction using a pretrained bert model @xcite . given the task of pruning a language dataset for fine-tuning, are influence scores useful signals for determining optimal data selection strategies? if so, which scores work best? we evaluate these scores against a random sampling baseline, in both noisy and clean data settings. user speech patterns are constantly evolving due to current events as well as user-system interactions that can be difficult to anticipate. are influence scores still effective in surfacing data critical for model performance in this dynamic setting? to answer this question, we build upon on our initial findings on snli and implement one influence score ("variance of gradients" or "vog", first presented in @xcite ) in a generic, large-scale nlu model stack commonly found in commercial voice assistants. we present results for existing in-house test data as well as results for a live user study in which we leveraged vog scores for the purpose of substantially reducing training data without incurring model-performance degradation. among the five influence scores we evaluated on snli, most out-of-the-box implementations do not beat a baseline of randomly pruning the dataset. the implementations can be improved to do better than the random-pruning baseline, but this typically requires careful experimentation to tune hyperparameters specific to each score. out of the scores we tested, we find that vog performs best relative to the random-pruning baseline, particularly at large pruning fractions. test accuracy is mostly maintained after pruning ∼45% of the snli training data using vog scores calculated in a "one-shot" fashion, i.e. from a single training run, without any score hyperparameter tuning. in a large-scale user study performed using the nlu stack, we find that sampling by vog scores is effective at surfacing training data that is particularly efficient for learning. we prune roughly 50% of training data without incurring statistically significant regressions in key metrics that track nlu errors, relative to a baseline model trained with all data. ## discussion in this work, we initiated a study of the application of influence scores in efficient language data sampling. first, we benchmarked a diverse subset of influence scores in a data reduction task and found promising results pruning using vog scores. in the second part, we used these preliminary results on snli as impetus to scale vog up to a model stack commonly used in commercial voice assistants. we provided a detailed account of how score normalization affects final results and again found encouraging results on experiments involving historical data using dataset-normalized vog scores, as well as in a user study. in particular, we did not see any overall regressions when a model trained only on ∼50% of historical data was deployed. this work mainly focused on data reduction; it would be interesting to reverse some of the presented analysis for data mixing/augmentation in order to identify economical ways of surfacing new model data. our work also focused on supervised settings using bert architectures; an obvious path forward would be to extend the definition of these scores to model pretraining and to, e.g., decoderonly architectures that are commonly used for large language models (see, e.g., @xcite along similar lines). while this may be difficult to implement at a microscopic level for a corpus of pretraining data such as the common crawl, one avenue could be to apply this method at a coarsegrained level by grouping together texts by similarity. given the recent results of @xcite and @xcite , this suggests a path towards training data-efficient large language models that could, in principle, outperform empirically observed scaling laws @xcite . another highly promising application of our results is determining the influence of specific examples in in-context learning. one concrete generalization of vog scores to this setting would be to look at variance of model weights (e.g., in attention heads) in specific layers over the length of the input sequence. this could provide an interpretable metric for identifying influential contextual examples and failure modes, at the level of specific tokens @xcite propose similar methods using influence functions). given the increased recent interest in this area of research due to concerns over bias, toxicity, and fairness in large language models, there is a critical need for simple, inexpen-sive, and empirical metrics that can estimate the the influence of examples to in-context learning. our work develops the foundational understanding necessary to make progress on that problem by generalizing results from the computer vision field (such as those scores that approximate more computationally expensive influence functions) to language-based tasks.