Self-Influence Guided Data Reweighting for Language Model Pre-training (2023)

Abstract: Language Models (LMs) pre-trained with selfsupervision on large text corpora have become the default starting point for developing models for various NLP tasks. Once the pre-training corpus has been assembled, all data samples in the corpus are treated with equal importance during LM pre-training. However, due to varying levels of relevance and quality of data, equal importance to all the data samples may not be the optimal choice. While data reweighting has been explored in the context of task-specific supervised learning and LM fine-tuning, model-driven reweighting for pretraining data has not been explored. We fill this important gap and propose PRESENCE, a method for jointly reweighting samples by leveraging self-influence (SI) scores as an indicator of sample importance and pre-training. PRESENCE promotes novelty and stability for model pre-training. Through extensive analysis spanning multiple model sizes, datasets, and tasks, we present PRESENCE as an important first step in the research direction of sample reweighting for pre-training language models.

## introduction language models (lm), typically pre-trained on large volumes of unlabeled text data, have become ubiquitous model choices for various challenging downstream tasks @xcite . the fundamental direction pursued for improving language model pre-training involves increasing the amount of training data or scaling model size @xcite . the training data is generally assembled from scraping the web and filtered using manually crafted heuristics that often require domain expertise @xcite . a key similarity in these prior works is the uniform treatment of the data samples available in the assembled corpora, without any regard for the data quality. ## background: tracin and self-influence though presence can be used with any influence function, we use tracin @xcite based self-influence score due to its scalability, generalizability, and effectiveness in identifying outliers. self-influence using tracin: tracin computes influence, i.e., how the loss on the test point changes during the training process whenever the training sample of interest was utilized by a first- si score 0.02 0.03 0.04 0.05 2 0.01 0.02 0.02 0.03 0.25 english hindi bengali clean text noisy text in-domain physics chemistry figure 1: left -average si scores across english, hindi, and bengali for clean and noisy (jumbled) text; right -average si scores for randomly sampled and domain mismatched information in english. we observe high si scores on average for noisy and domain mismatched text. more discussions in section 2.1. self-influence score measures the influence a sample has on itself. this is identical to replacing z ′ with z in equation 1, giving tracinsi(f θ , •) as, ## presence-sequential: filtering pre-training data using self-influence extending tracin based self-influence for pretraining: as pre-training is computationally expensive, we leverage the layer agnostic nature of tracin and introduce an optimized layer-wise self-influence calculation. for layers @xmath0, k 2 , . . . , k k } of model f θ , let f θ,k denote the parameters of layer k. self-influence for any layer set k ⊂ k is, as shown in section 2.1, there is a relation between si scores and the sample quality. we leverage this property to filter large-scale web corpora in an offline manner to create more suitable pretraining data. we present an overview of our offline sequential filtering strategy using self-influence scores, called presence-sequential, in figure 2 . assuming that a model requires n training samples for pre-training, we choose n samples from a set of n ′ > n samples by filtering out samples with the highest si scores using a proxy model trained we sort tracinsi k (f θ , d ′ ) in increasing order of si scores and filter out n ′ -n samples with the highest score. the remaining n samples comprise the filtered set d used for pre-training, pre-training setup: we use the mc4 dataset @xcite and pre-train an mt5-base model for 200, 000 steps on randomly shuffled data, and use this as the 'scoring model (f θ )' to create the we use the first layer of the encoder and the first layer of the decoder in the set k for tracinsi k . downstream tasks and fine-tuning: following @xcite , we utilize datasets across 5 tasks from the xtreme multilingual benchmark @xcite , including question answering, sentence-pair, and structured prediction. we evaluate on (i) zero-shot cross-lingual transfer: where the fine-tuning data is only in english, and (ii) translate-train: where the fine-tuning data is in english and translated into the target languages for all the downstream datasets. we summarize the datasets used for evaluation in ## presence: sample reweighting using self-influence in this approach, we use the si scores in an online joint setting by reweighting samples at the minibatch level. we calculate sample si scores at each training step and use them to weigh sample gradients before aggregating them for the gradient update. to formulate sample reweighting using si scores, we consider batch @xmath1, n]}, where z i denotes a sample. we calculate si scores using equation 3 for each sample to get array s, where |s| = n, training loss training steps τ > 0 τ < 0 stage 1: direct weighting stage 2: inverse weighting where µ(•) and σ 2 (•) denote the mean and variance, respectively, and ϵ is a small number. to calculate relative weights for the samples' gradients, we use a softmax function, softmax(•) over each s i with temperature τ to get weights w i , here, w i gives the weight for the gradient of sample z i . using weights w i , the gradient g for the model update is given by, si scores to limit the impact of noisy and unreliable samples. two-staged reweighting ensures that the model learns novel information early, and is able to eliminate noise at a later stage with stable learning. for temperatures τ 1 > 0 and τ 2 < 0, the softmax temperature at training step i is given by, ## pre-training adaptation pre-training requires a large batch size and is prone to instability @xcite . we thus adapt presence for pre-training by applying it at the microbatch level. this provides dual benefits of regularizing the pre-training while being computationally efficient. ## related work datasets for pre-training lms language models are generally pre-trained on large-scale corpora scraped from the web @xcite @xcite . the most common source of obtaining large-scale data is common crawl foot_2 , a publicly-available web archive that provides "web extracted text". raffel et al. (2020) use various heuristics such as retaining lines that end in terminal punctuations, retaining lines and pages based on a minimum number of words, etc to clean common crawl. wenzek et al. (2020) use a kneser-ney language model @xcite and calculate the perplexity over training data and a high quality target domain to extract high quality documents. multilingual pre-training has also known to depend on language and domain distributions in the corpora @xcite @xcite . multilingual pre-training involves an additional step of boosting low-resource language data in the corpora using temperature sampling @xcite @xcite . doremi (xie et al., 2023) uses a smaller proxy model to calculate domain weights of different sources comprising the mixture of the pre-training data to pretrain larger models. these works either rely on expertly crafted heuristics or require training additional models for dataset selection and filtering. influence functions and training data attribution influence functions help to trace a model's prediction through the learning algorithm and back to its training data, a practice commonly known as training data attribution (tda) @xcite . influence functions have been extensively used in deep learning as a means of model interpretability and explainability @xcite @xcite , adversarial learning @xcite , federated learning @xcite , and identifying outliers or mislabeled samples @xcite . tracin @xcite introduces a scalable and general first-order approximation to calculate gradient based influence, and extends it to the minibatch level. @xcite formulates an automated curricular strategy using si scores for data cleaning and filtering for nlp tasks. influence functions have mostly been applied for supervised learning with ground truth labels for the data and have generally been explored in an offline setting. data selection and online adaptation in supervised learning selection functions for supervised learning often leverage training dynamics such as high loss @xcite or high prediction uncertainty @xcite to select "hard" points. @xcite use the change in loss over the course of training rather than each step to also eliminate noisy samples. removing noisy training samples using offline methods is another direction for selecting training data for supervised learning @xcite . paul et al. (2021) use norm of the gradient or self-influence to identify important samples early in the training to heavily prune datasets. rho-loss (mindermann et al., 2022) calculates a heldout loss using a proxy model and uses a combination of model loss and heldout loss to select non-noisy, non-redundant, and taskrelevant samples. ahn et al. (2023) uses per-sample gradient norm to assign importance probabilities, and trains a biased model to formulate a debiased model training strategy. ren et al. (2018) and (jain and shenoy, 2022) use meta-learning for reweighting training samples within a batch for increasing robustness and selective prediction respectively. these works operate in supervised settings, requiring controlled validation sets or proxy models and adapting them to pre-training is non-trivial. ## conclusion and future work we introduce presence -a method for jointly reweighting samples using self-influence (si) scores and pre-training. we conduct an in-depth analysis of the relationship between si scores and sample quality from a pre-training perspective and use them as a filtering objective for pre-training data selection. as sequential filtering is expensive at the scale of pre-training, we formulate pres-ence as a joint adaptation for sample reweighting. presence outperforms baselines trained on randomly sampled and si-filtered data on 5 datasets across 3 tasks. we believe that presence is an important first step in the research direction of data sample weighting for pre-training. as future work, we plan to explore relationships between samples in the pre-training corpora and influence functions across languages, data sources, and domains. we also plan to formulate automated reweighting strategies using temperature scaling schedules based on the training step, training loss, and sample influence scores.