Language models and brains align due to more than next-word prediction and word-level information (2024)

Abstract: Pretrained language models have been shown to significantly predict brain recordings of people comprehending language. Recent work suggests that the prediction of the next word is a key mechanism that contributes to this alignment. What is not yet understood is whether prediction of the next word is necessary for this observed alignment or simply sufficient, and whether there are other shared mechanisms or information that are similarly important. In this work, we take a step towards understanding the reasons for brain alignment via two simple perturbations in popular pretrained language models. These perturbations help us design contrasts that can control for different types of information. By contrasting the brain alignment of these differently perturbed models, we show that improvements in alignment with brain recordings are due to more than improvements in next-word prediction and word-level information.

## introduction language models (lms) that have been pretrained to predict the next word over billions of text documents have also been shown to significantly predict brain recordings of people comprehending language @xcite @xcite @xcite . understanding the reasons behind the observed similarities between representations of language in machines and representations of language in the brain can lead to more insight into both systems. recent studies suggest that the prediction of the next word is a key mechanism that contributes to the alignment between the two @xcite . what is not yet understood is whether prediction of the next word is necessary for this ob-code available at github.com/gab709/brain-llm-beyond-next-word. figure 1 : an illustration of additional information that may be important for alignment between language models and brain recordings. our approach is largely agnostic about the exact linguistic information contained in the conceptual quantities "word-level information" and "multi-word information", and the only assumption is that "word-level information" is not affected by word order. understanding the impact of other kinds of information on brain alignment is complicated by correlations with next-word prediction (nwp). because nwp is the lm training objective, better nwp may also be related to improved representations of other types of information @xcite that the human brain is sensitive to, such as word-level or multi-word information @xcite . neuroscientists are still investigating the exact linguistic features at the word-and multiword levels that are important for processing in the brain, so for the remainder of the paper, we take an abstract approach and refer to "word-level information" as the non-contextualized representation of the word, and to "multi-word information" as relating to multiple words (e.g. syntax). for example, each word in "harry throws the broom" has a non-contextualized meaning and the phrase has a different meaning depending on the word order ("harry throws the broom" vs. "the broom throws harry", see figure 1 ). we note that these are conceptual quantities and not ones we are claiming to be able to quantify directly. the only assumption key to our argument is that word-level information is not impacted by word order. other than that, our methods are agnostic about the specific linguistic information contained by these conceptual quantities. both word-level and multi-word information may contribute to brain alignment, but their effect cannot be disentangled from that of next-word prediction using previous approaches. in this work, we aim to disentangle the contributions of next-word prediction and word-level information from other factors, such as multi-word information, in the brain alignment of gpt-2-based models @xcite . our methodology builds upon the traditional neuroscientific approach of constructing contrasts between brain activity elicited by different conditions. a contrast reveals the processing of a specific property p in the brain, by subtracting the brain activity elicited by two conditions (condition a and b) that are tightly controlled to contain similar information except for the target property p. we leverage this approach and design contrasts between the predicted brain activity by two related models: an original model and its perturbed version. by contrasting the brain alignment of these two models, we can conclude that any difference in brain alignment is due to the perturbation. conversely, if the two contrasted conditions are controlled for some factor, then any difference in the predicted brain activity between the two conditions cannot be due to this factor. our key insight is to design a contrast that controls for both information related to next-word prediction and word-level information. this contrast is enabled by two proposed perturbations. the first perturbation, which we name input scrambling, scrambles the order of the input words at inference time. this perturbation controls for the wordlevel information when contrasting a model's brain alignment related to the original vs. the scrambled inputs, because, by definition, the word-level information encoded in the model representations remains the same. any remaining brain alignment after the contrast must therefore be due to factors beyond word-level information, such as next-word prediction or multi-word information. the second perturbation further disentangles the contribution of next-word prediction to brain alignment. this perturbation, which we name stimulus-tuning, finetunes a model to predict the next word in the specific naturalistic stimulus text corresponding to the brain recordings. the fine-tuning is done until the next-word prediction performance matches a pre-defined level above its input-scrambled version, similar to the baseline model's improvement over its input-scrambled version. contrasting brain predictions from pairs of models--baseline vs. scrambled and stimulus-tuned vs. scrambled--controls for both word-level information and next-word prediction. any residual brain alignment is then due to other factors, such as multi-word information. we note that our methodology is based on models trained to predict the next word; however, we only control for next-word prediction performance at inference time. therefore, we cannot draw conclusions about the importance of the training objective itself, but rather about the information that has emerged after the training process. after controlling for word-level and next-word prediction in the final contrast, we still observe residual brain alignment. across three types of models (gpt-2-small, gpt-2-medium @xcite and gpt-2-distilled @xcite ), we find consistent residual brain alignment in two specific brain areas that are thought to process language @xcite )-the inferior frontal gyrus (ifg) and the angular gyrus (ag)suggesting that the brain alignment between the language model and these brain regions is due to more than next-word prediction and word-level information. we speculate that this alignment is due to multi-word information, which is consistent with previous findings about processing in these regions @xcite . our main contributions are as follows: (i) propose perturbations to pretrained language models that, when combined in suitable contrasts, can control for the effects of next-word prediction and word-level information on brain alignment; (ii) demonstrate that a proposed perturbation, which consist of tuning a language model on a validation stimulus text, can increase the alignment with brain recordings that correspond to a heldout text; (iii) reveal that the brain alignment with language regions, in particular in the inferior frontal gyrus (ifg) and the angular gyrus (ag), is due to more than nextword prediction and word-level information. ## results in this section, we report the results averaged across models. the results for the individual models can be found in appendices h, i, j. ## related works several previous studies have investigated the alignment between pretrained language models and brain recordings of people comprehending language, finding significant similarities @xcite @xcite @xcite . our work builds on these and further studies the reasons for these similarities. the work of @xcite is most directly related to our research question, as they suggest that the prediction of the next word is a key reason for the alignment between language models and brain recordings, based on evidence that ecog electrodes can predict the neural network representation of upcoming words in a story. our work uses perturbations to disentangle next-word prediction from other types of information that may affect brain alignment (word-level and multi-word information) and offers an additional account of the necessary information for brain alignment. our work also relates to a growing body of research on disentangling the contributions of different types of information to the alignment between brain recordings and language models. toneva et al. (2022a) present an approach to disentangle supra-word meaning from lexical meaning showing that the supra-word meaning is predictive of fmri recordings in two language regions (anterior and posterior temporal lobes), which was further adapted by oota et al. (2024a) and oota et al. (2024b) to disentangling effects of other linguistic properties. caucheteux et al. (2021) and reddy and wehbe (2021) aim to disentangle alignment due to syntactic and semantic processing. toneva et al. (2022b) examine whether representations obtained from a language model align with different language processing regions in similar or different ways. kauf et al. (2023) investigate the contribution of word-level semantics to the brain alignment of language models, showing that syntactic perturbations have a lesser impact on brain alignment compared to semantic perturbations. gauthier and levy (2019) demonstrate that fine-tuning language models on scrambled data has been shown to be beneficial for brain decoding. our experiments reveal that the scrambling perturbation influences brain alignment. despite this, when sentences are scrambled at inference time, the model retains the capability to predict brain responses in regions as-sociated with language processing. however, the aim of our work is not to directly evaluate the significance of semantics or syntax, but rather to investigate the effects on brain alignment when information relevant to next word prediction is controlled for. therefore, our proposed perturbations are complementary to these previous works and may yield additional insights if combined. other studies have used perturbations related to word order to investigate some properties of language models. @xcite introduced distracting content to test how robustly language models retain and use that information for prediction, showing that language models are particularly susceptible to semantic similarity and word position. @xcite applied a perturbation (scrambling method) to investigate where the semantic and syntactic processing is taking place in bert, revealing that early layers care more about the lexicon, while the latter layer care more about word order. our current work contributes to this research direction by examining the effects of scrambling on both brain alignment and language modeling. finally, a work by aw and toneva (2023) finetunes language models to summarize narratives and finds improved brain alignment, despite a lack of improvement in next-word prediction. while this finding suggests a similar conclusion to the one from our work-that next-word prediction performance is not necessary for improved brain alignment-the perturbation approach in our work allows additional control over the language model representations and is complementary to this previous work. ## discussion we showed that the perturbation that we termed stimulus-tuning (i.e., finetuning a pretrained model on a validation stimulus text) can increase the alignment with brain recordings that correspond to a heldout text, particularly in several language processing brain areas. we quantified this improvement by comparing the stimulus-tuned model and the baseline in these brain areas. stimulus-tuning may improve brain alignment due to improved ability to represent the next word, previously seen individual words, or multi-word information that are specific to the stimulus narrative. using the perturbation that we termed input scrambling, we showed that the improved next-word prediction capabilities of the stimulus-tuned model is not the only reason for improved brain alignment. we showed that leveraging a contrast that controls for word-level information and nextword prediction, we still obtain a residual brain alignment. specifically, we show that, across multiple gpt-2 models, improvements in alignment with brain recordings in two language regions-inferior frontal gyrus (ifg), angular gyrus (ag)-(see figure 5 ) are due to more than improvements in next-word prediction and word-level information. one possible reason for this improvement in brain alignment is improved capabilities to represent multi-word information that are specific to the stimulus text. this hypothesis aligns with previous work that has found the inferior frontal gyrus (ifg) to be sensitive to syntax @xcite and the angular gyrus (ag) to multi-word event structure @xcite . note that the fact that we do not find strong effects in other language regions does not necessarily mean that they do not process multi-word information. ## conclusion this work aims to deepen our understanding of the existing alignment between language models and brain recordings. we proposed two perturbations to pretrained language models that, when used together, can control for the effects of next-word prediction and word-level information on the alignment with brain recordings. using these controls, we show that improvements in brain alignment are due to more than improvements in next-word prediction and word-level information. our findings are relevant for both cognitive neuroscientists as well as natural language processing researchers. the findings are relevant for cognitive neuroscientists because they suggest that accurate prediction of the next word is not a necessary condition for brain alignment. it is possible that learning to accurately predict the next word is sufficient for inducing other properties in the language model representations that are necessary for brain alignment, such as multi-word information, and future work can further examine this hypothesis. our findings are also relevant to nlp researchers who examine what language models can learn from only text. we show that finetuning a language model with small amounts of text can increase its alignment with never-before-seen brain recordings, and that this improvement in brain alignment is not purely due to next-word prediction or word-level information. this finding suggests that training a language model with little additional text can improve its representations of multi-word information in a brain-relevant way. we note that while our methodology controls for next-word prediction ability at inference time, it still relies on the nextword prediction objective during training. however future work can investigate alternative training objectives, that may improve their ability to represent multi-word information in other ways. one example is work by aw and toneva (2023) that shows that finetuning a language model using a summarization objective can further improve brain alignment.