Learning to vary: Teaching LM s to reproduce human linguistic variability in next-word prediction (2025)

Abstract: Natural language generation (NLG) tasks are often subject to inherent variability; e.g. predicting the next word given a context has multiple valid responses, evident when asking multiple humans to complete the task. While having language models (LMs) that are aligned pluralistically, so that they are able to reproduce well the inherent diversity in perspectives of an entire population of interest is clearly beneficial, Ilia and Aziz (2024) show that LMs do not reproduce this type of linguistic variability well. They speculate this inability might stem from the lack of consistent training of LMs with data reflecting this type of inherent variability. As such, we investigate whether training LMs on multiple plausible word continuations per context can improve their ability to reproduce human linguistic variability for next-word prediction. We employ fine-tuning techniques for pre-trained and instruction-tuned models; and demonstrate their potential when fine-tuning GPT-2 and Mistral-7B-IT, using Provo Corpus. Our evaluation, which measures divergence among empirically estimated human and model next-word distributions across contexts before and after fine-tuning, shows that our multi-label fine-tuning improves the LMs’ ability to reproduce linguistic variability; both for contexts that admit higher and lower variability.

## introduction inherent variability in natural language generation (nlg) tasks might arise from ambiguity or varying perspectives @xcite . for example, when predicting the next word given a context, multiple plausible and valid continuations exist; a task whose linguistic variability we can appreciate by asking a human population to complete it @xcite . we can also appreciate this type of linguistic variability for autoregressive language models (lms) that generate text by sampling from next-token (i.e. subword unit) distributions conditioned on preceding tokens @xcite . we achieve that by viewing such distributions as a representation of the model's uncertainty over continuations given a prefix @xcite @xcite . it is often valuable for models to reproduce such variability, particularly in openended nlg tasks, where multiple responses can be plausible. whereas this variability contributes to making lms more robust @xcite @xcite and more representative of the linguistic diversity of human populations of interest @xcite , it has been shown that the variability lms exhibit does not always align with the one humans exhibit @xcite @xcite . for next word prediction, @xcite identify this misalignment and speculate it might stem from inconsistent exposure of lms to training data reflecting such variability. as such, we investigate whether training lms with multiple observations of the next word per context will improve their ability to reproduce human variability. while previous fine-tuning work utilising multiple references per instance focused on classification tasks @xcite @xcite , our work focuses on next-word prediction, a generative task. similar to @xcite , who employ a form of multi-label distillation in next word prediction, we also employ a technique to fine-tune pre-trained lms and extend to instruction-tuned lms. for the former, we alter the training signal, and for the latter we exploit a training data augmentation method to ensure that variability is observed. we employ these fine-tuning techniques for gpt-2 @xcite , a pre-trained model, and mistral-7b-it @xcite , an instructiontuned model. when evaluating, by measuring divergence among empirically estimated human and model next-word distributions across contexts, be-fore and after fine-tuning, we find that fine-tuning with multiple labels per instance improves those lms' ability to reproduce linguistic variability, across contexts of varying open-endedness. additional ablations measure performance when varying the number of training labels per instance; and with a preliminary analysis we measure the tradeoff in performance in tasks that admit no plausible variability. for that, we handcraft a small evaluation dataset using a knowledge-based question answering dataset @xcite . foot_0 2 related work human label variation in natural language processing (nlp) tasks is often dismissed as noise @xcite . however, multiple responses can be plausible, especially relevant to ambiguous or open-ended tasks or prompts @xcite @xcite @xcite . embracing this plausible variation as part of nlp systems, which could make them more fair @xcite and robust @xcite , involves altering all stages of our systems' development pipelines: from dataset creation, collecting multiple labels per prompt @xcite , to training, utilising these labels during the learning phase @xcite @xcite , and evaluation, comparing models' responses to multiple human references @xcite . our approach aims to embrace plausible variability during training. rather than collapsing annotations into a single ground truth @xcite , we incorporate multiple plausible references. the idea of multi-label fine-tuning has been adopted in image-classification @xcite @xcite , as well as in nlp, primarily for classification @xcite @xcite @xcite @xcite . additionally, recent efforts have applied instruction fine-tuning for multi-label text classification tasks @xcite and tasks with restricted outcome spaces, such as sampling from discrete distributions @xcite . our work focuses on a generative task, (i.e., that of predicting ## methodology we exploit simple yet intuitive fine-tuning techniques, depending on the lms' previous training. these require a set of contexts @xmath0, ..., c n }, where for each context c i , we have a set of human next-word references w @xmath1, ..., w im } foot_1 : fine-tuning pre-trained lms autoregressive lms are trained using cross-entropy between a target and the model's next-token distribution given context c (p(•|c) and q(•|c) resp.). this corresponds to searching for the maximum likelihood estimate (mle). when training on a corpus with a single continuation (i.e. the next corpus token w * ), p is a deterministic distribution centered on w * , leading to the following loss: when multiple word continuations are available, we replace this deterministic distribution with the empirically estimated distribution (using w i ), where the probability of a word given c i , p(w|c i ), equals its relative frequency in w i . this results in the following loss, which comprises generalized cross entropy @xcite : fine-tuning instruction-tuned lms instructiontuned models underwent additional training to cater a rather conversational format and adhere to task instructions. we sample responses from the model's conditional predictive distribution (cpd) given a prompt, i.e. an instruction and an example. for our task, we sample response r containing a predicted word from q(r|(i, c i )), where the prompt includes instruction i requesting a word continuation given a prefix, and the example context c i . so as to utilise multiple labels, we employ the following training data augmentation technique: for each context c i in c, we construct the prompt (i, c i ) and for each word w j in w i , we create a training datapoint where w j is a response to (i, c i ). this entails that c i will appear multiple times with different continuations as per their frequency in w i . we train using l label . 5 see appendx a for prompts. 2023 )), which we refer to as mistral-7b-it. both models are fine-tuned using provo corpus (luke and christianson, 2018), which contains 55 text passages (2687 total contexts). each prefix is annotated with an average of 40 human annotations predicting the word following it. we split the dataset randomly at the paragraph level (to avoid partial passage leaks between train and test sets). 80% is for training, of which 10% is reserved for validation; and the remaining for testing. training configuration. both models were finetuned using the adam optimizer @xcite . for gpt-2: we train for 3 epochs, using a learning rate of 1e -5 and a batch size of 16. for mistral-7b-it: we train for 4 epochs using low-rank adaptation @xcite with a learning rate of 1e -4 with a batch size of 32. we train on 3 random seeds; training details in appendix b. metrics. following ilia and aziz (2024): for each context, we measure the divergence between the human and model cpds given a context using total variation distance (tvd) (rudin, 1987). 6 tvd quantifies the difference between two probability distributions by summing the absolute differences in the probabilities they assign to the same event. a higher tvd indicates greater disagreement between human and model cpds (i.e., poorer alignment with human linguistic variability), whereas a lower tvd indicates less disagreement (i.e. better alignment with human linguistic variability). in order to compute tvd, we need estimates of the human and model cpds (p(w|c) and q(w|c) respectively). as done in ilia and aziz (2024): (1) for p(w|c), we estimate it via monte carlo, with p(w|c) equaling the relative frequency of w in all human samples, and (2) for q(w|c), we estimate it via monte carlo, by sampling 40 sequences from the model long enough to contain a full word, slice it, and compute q(w|c) (or q(w|(i, c))) as the relative frequency of w in all sampled words. baselines & upper bounds. we compare the distribution of tvd values across contexts before and after fine-tuning, where improved performance would mean a shift towards lower tvd values (i.e. less disagreement with human cpds). for the instruction-tuned model, we add a 1-shot baseline, where the prompt includes an example of a context and word references (details in appendix a). as another baseline, we fine tune models with provo's original corpus passages (i.e. one continuation per prefix), imitating models' usual training. lastly, to estimate the best performance we can expect from our models, which essentially is to mimic human divergence, we establish a baseline for the expected level of disagreement from humans for a context. we split human responses in two disjoint groups and measure their cpds' tvd ('oracle' baseline). ## conclusion this study examines whether fine-tuning with multiple labels per instance has the potential to enhance models' ability to reproduce linguistic variability in next word prediction. we show improved performance for a smaller pre-trained language model (gpt-2) and a larger instruction-tuned model (mistral-7b-it) across contexts that admit varying levels of plausible variability. our findings highlight both the potential and possible limitations of such fine-tuning, paving the way for further advancements in modeling linguistic variation.