Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects (2021)

Abstract: A popular approach to decompose the neural bases of language consists in correlating, across individuals, the brain responses to different stimuli (e.g. regular speech versus scrambled words, sentences, or paragraphs). Although successful, this ‘model-free’ approach necessitates the acquisition of a large and costly set of neuroimaging data. Here, we show that a model-based approach can reach equivalent results within subjects exposed to natural stimuli. We capitalize on the recently-discovered similarities between deep language models and the human brain to compute the mapping between i) the brain responses to regular speech and ii) the activations of deep language models elicited by modified stimuli (e.g. scrambled words, sentences, or paragraphs). Our model-based approach successfully replicates the seminal study of Lerner et al. (2011), which revealed the hierarchy of language areas by comparing the functional-magnetic resonance imaging (fMRI) of seven subjects listening to 7min of both regular and scrambled narratives. We further extend and precise these results to the brain signals of 305 individuals listening to 4.1 hours of narrated stories. Overall, this study paves the way for efficient and flexible analyses of the brain bases of language.

## introduction one of the most successful paradigms to decompose the brain bases of language consists in correlating the brain responses of multiple subjects listening to the same carefully controlled stimuli @xcite @xcite . in particular, @xcite recorded subjects with functional magnetic resonance imaging (fmri) while they listened to a story whose (1) sounds (2) words, (3) sentences or (4) paragraphs were scrambled, as well as ( 5 ) to the regular version of the story (figure 1a ). the authors then estimated the inter subject correlation (isc), i.e., the correlation between i) the brain activity of a voxel in response to one scrambling condition and ii) the brain activity of a voxel averaged across all other subjects, in response to the same scrambled stimulus (figure 1b ). while successful, this 'model-free' approach is costly: it requires n subjects × n conditions acquisitions of brain activity in response to the same variably scrambled stimuli. here, we investigate whether and how a modelbased approach can replicate lerner et al.'s findings, even if we only have access to the recordings elicited by the regular story in a single subject. we further apply the method to extend lerner et al's results to a large dataset of 305 individuals. ## methods first, we formalize the 'model-free' and 'modelbased' approaches in the context of narrative listening, and explicit the link between the two. ## experiment to test our model-based approach, we first apply it to the fmri responses of 75 subjects listening to the same 7 min story analysed in @xcite foot_0 . thus, for each condition (word, sentence and paragraph), subject and voxel, we compute the model-to-brain correlation @xmath0, f θ (x * ) . the extraction of the fmri signals y, and the estimation of the mapping function f θ are standard and thus detailed in appendices a and c. to estimate context-free representations, we i) scramble the stimulus at the word, sentence or paragraph level, ii) extract the corresponding activations x from a deep language model, and iii) compute x * , as detailed below. scrambling the stimulus at the word, sentence and paragraph level words and sentences of the stimulus are delimited using spacy tokenizer (honnibal et al., 2020). note that punctuation marks are not considered as words (e.g., 'time.' forms one token, not two). we define paragraphs as contiguous chunks of eight sentences. to 'scramble' a sequence at the word (resp. sentence, paragraph) level, we uniformly shuffle the indices of its words (resp. sentences, paragraphs) and form the new sequence accordingly. ## results the results are displayed in figure 2b . the hierarchy of temporal receptive fields (trfs) typically associated with acoustic, word, sentence and paragraph processing along the temporo-parietal axis is remarkably well replicated in both hemispheres (figure 2b ). notably, both the model-free and model-based methods evidence that the precuneus, the superior frontal gyrus and sulcus are characterized by sentence-and paragraph-level trfs (figure 2a and b ). our results differ from lerner et al.'s in several ways. first, the acoustic trfs are slightly more inferior with the model-based method. second, frontal regions are detected to be associated not only with sentences and paragraphs, but also with words (consistent with @xcite @xcite ). given that lerner et al's dataset is not public, it is difficult to quantify these differences and determine whether they reflect an improved sensitivity, or, more simply, inter-individual differences. our model-based method can, in principle, be applied to any natural stories. to test this prediction, we extend our analyses to 305 subjects listening to 4.1 hours of fifteen narratives (figure 2c ). our model-based approach recovers the hierarchy of trfs, and further reveals additional word-and sentence-level representations in the precuneus and prefrontal regions. ## discussion here, we leverage the modeling power of deep language models to show that the seminal results of lerner et al. can be retrieved without having subjects listening to multiple scrambled stimuli. critically, we formalize the assumptions under which 'model-based' and 'model-free' approaches can be linked @xcite . our model-based method recovers the hierarchy of trfs evidenced by @xcite in the brain of an unusually large cohort of 305 subjects. thus, our study complements the recent work of @xcite @xcite who predict brain responses to speech from language models input with variably-long con-texts. specifically, we show that previous modelbased results unravel the same mechanisms that was previously identified with model-free approaches. the replication is not perfect: the acoustic and word trfs slightly differ between the two methods. this may be explained by individual subject's variability, which is only captured by the model-based approach. further research, using the non-public data from lerner et al. should investigate these remaining differences. in line with previous work @xcite @xcite , our study demonstrates that deep neural networks build constructs that predict brain activity, accurately enough to recover the hierarchy of language processing in the brain. the success of replication thus reinforces the idea that naturalistic stimuli and deep neural networks form a powerful couple to study the neural bases of language (hamilton and huth, 2020). 2016) using the gpt-2 tokenizer provided by huggingface. then, we feed the network with the m bpe tokens (@xmath1, up to 256 tokens in memory) and extract the corresponding activations from layer l, of shape (m × d) with @xmath2. then, we sum the activations over the bpes of each word to obtain a vector of size (m × d). all our analyses are based on the eighth layer of gpt-2. we choose gpt-2 because it has been shown to best encode the brain activity elicited by language stimuli @xcite . we choose its eighth layer because the intermediate layers of transformers have shown to encode relevant linguistic features @xcite and to better encode brain activity than input and output layers @xcite . scrambling the stimulus at the word, sentence and paragraph level words and sentences of the stimulus are delimited using spacy tokenizer @xcite . note that punctuation marks are not considered as words (e.g., 'time.' forms one token, not two). we define paragraphs as contiguous chunks of eight sentences. to 'scramble' a sequence at the word (resp. sentence, paragraph) level, we uniformly shuffle the indices of its words (resp. sentences, paragraphs) and form the new sequence accordingly.