The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Units (2025)

Abstract: Large language models (LLMs) exhibit remarkable capabilities on not just language tasks, but also various tasks that are not linguistic in nature, such as logical reasoning and social inference. In the human brain, neuroscience has identified a core language system that selectively and causally supports language processing. We here ask whether similar specialization for language emerges in LLMs. We identify language-selective units within 18 popular LLMs, using the same localization approach that is used in neuroscience. We then establish the causal role of these units by demonstrating that ablating LLM language-selective units – but not random units – leads to drastic deficits in language tasks. Correspondingly, language-selective LLM units are more aligned to brain recordings from the human language system than random units. Finally, we investigate whether our localization method extends to other cognitive domains: while we find specialized networks in some LLMs for reasoning and social capabilities, there are substantial differences among models. These findings provide functional and causal evidence for specialization in large language models, and highlight parallels with the functional organization in the brain.

## introduction recent advancements in large language models (llms) have revealed their potential to perform far more than language processing tasks, showcasing abilities in reasoning @xcite , problem-solving @xcite , and even mimicking aspects of human theory of mind @xcite . despite these impressive feats, the internal workings of llms remain poorly understood, especially in relation to how ## preliminaries the human language network. the human language network comprises a set of brain regions that are functionally defined by their increased activity to language inputs over perceptually matched controls (e.g., lists of non-words) @xcite . these regions are predominantly localized in the left hemisphere, within frontal and temporal areas, and demonstrate a strong selectivity for language processing over various non-linguistic tasks such as music perception @xcite and arithmetic computation @xcite . crucially, these regions exhibit only weak activation in response to meaningless non-word stimuli, whether during comprehension or production @xcite . this high degree of selectivity is well-established through neuroimaging studies and is further supported by behavioral data from aphasia studies: in individuals with damage confined to these language areas, linguistic abilities are significantly impaired, while other cognitive functions-such as arithmetic computations @xcite , general reasoning @xcite , and theory of mind @xcite -remain largely intact. in addition to language-specific systems, the brain supports higher-level cognition through distinct networks that handle demanding tasks and social reasoning. the multiple demand network. the multiple demand network (md), encompassing bilateral frontal, parietal, and temporal regions, is activated during cognitively demanding tasks, showing a consistent "@xmath0.g., spatial, verbal, mathematical; @xcite @xcite . this network underpins key cognitive functions such as working memory, cognitive control, and attention, and is linked to fluid intelligence @xcite . the theory of mind network. the theory of mind (tom) network, primarily located in the bilateral temporo-parietal junction and cortical midline, is involved in reasoning about mental states-whether one's own or others' @xcite @xcite . functionally and anatomically distinct from the language network, the tom network is engaged across different content types (e.g., verbal, non-verbal) and is engaged in understanding non-literal language such as sarcasm, and for discourse comprehension where multiple perspectives need to be inferred @xcite . ## localizing the language network the human language network is defined functionally rather than anatomically which means that units are chosen according to a 'localizer' experiment @xcite . specifically, the language network is the set of neural units (e.g., voxels/electrodes) that are more selective to sentences over a perceptually-matched control condition (e.g., lists of nonwords) @xcite as illustrated in figure 1 . in previous studies comparing artificial models to brain activity in the language network, units were selected by evaluating representations at different model layers and choosing the ones that maximized brain alignment @xcite @xcite . however, llms learn diverse concepts and behaviors during their considerable pretraining, not all of which are necessarily related to language processing. therefore, we here characterize the language units in llms using functional lo-calization as is already standard in neuroscience. this approach comes with the advantage of comparability across different models since we can choose a fixed set of units which are localized independently of the critical experiment or modality. specifically, we present each llm with 240 unique 12-word-long sentences and 240 unique strings of 12 non-words used in neuroscience @xcite , ensuring matched sequence lengths across conditions. human participants are typically exposed to 96 sentences/non-word strings during an experimental fmri session @xcite . we then capture the activations from the units at the output of each transformer block for each stimulus. we define the model's language network as the top-k units that maximize the difference in activation magnitude between sentences and strings of non-words, measured by positive t-values from a welch's t-test. this localization method selects a targeted set of units across the entire network, rather than restricting the representations to a single a priori stage as done in prior work @xcite @xcite . we examine unit activations after each transformer block. for instance, for a model like llama-3-8b @xcite which consists of 32 transformer blocks and a hidden dimension of 4096, we consider 32 × 4096 = 131, 072 units, from which we select the most language selective units as the model's "language network". ## experimental setup models we selected 10 autoregressive llms from a diverse range of model families to investigate their language-selective units: gpt2-{large, xl} @xcite , llama-2-{7b, 13b} @xcite , llama-3.1-8b-instruct @xcite , mistral-7b-v0.3 (jiang et al., 2023), falcon-7b @xcite , phi-3.5-mini-instruct @xcite , and gemma-{2b, 7b} @xcite . the models were downloaded from the huggingface platform @xcite . language benchmarks to assess the significance of the localized units on the models' linguistic abilities, we utilize three widely used benchmarks that measure formal linguistic competence @xcite . first, syntaxgym (gauthe distribution of the top 1% most languageselective units across layers in a sample of five different models. the models are displayed from top to bottom, with each layer labeled by the percentage of units identified as belonging to the top 1% language-selective units. ## a specialized language network in llms figure 2 (a) shows the percentage of language units in each layer that belong to the top 1% of the most selective units for five models analyzed in this study (additional heatmaps for other models can be found in appendix b). figure 2(b) demonstrates a language selectivity index against the relative depth of each layer across all models tested. this index is calculated by summing 1p-values for each unit where @xmath1.05 after false discovery correction, and normalizing by the hidden dimension size. ## similarity between the language network in llms and brains qualitatively similar response profiles between the language network in llms and brains. in this analysis, we record the responses of the localized units to the exact stimuli from four experimental conditions used in neuroscientific studies @xcite , along with a set of non-linguistic stimuli such as arithmetic equations and code. this allows us to assess how well the selectivity of localized language units generalizes to new stimuli from the same conditions (sentences and strings of non-words) and how well they map onto results from neuroscience (amalric and dehaene, 2019; @xcite . stimuli are presented in four conditions (examples in figure 4a ): sentences , denoted as s, are well-formed sentences containing both lexical and syntactic information. unconnected words , w, are scrambled sentences containing lexical but not syntactic information. jabberwocky sentences , j, where content words are replaced by pronounceable non-words (such as "pront", or "blay"), thus containing syntactic but not lexical information. unconnected non-words , n, which are scrambled jabberwocky sentences containing neither lexical nor syntactic information. note that we use a disjoint set of sentences and non-words for the original functional localization (section 3). the brain's language regions are highly sensitive to linguistic structure: responses to s are numerically higher than all other conditions @xcite @xcite . the llm language units exhibit a similar response pattern to that of the brain's language network (figure 4c , first 4 bars). consistent with human neuroscience @xcite @xcite , llm language units are particularly selective for natural language compared to arithmetic equations, c++ code, or random sequences of characters (all matching the number of tokens and samples in * ** * n.s. ** n.s. ** n.s. * n.s. n.s. n.s. figure 5 : language units are aligned to brain data. raw pearson correlation between predicted brain activity from the x% of model units and actual brain activity in the human language network across 10 models. the alignment of language-selective units shows significantly greater correlation compared to the average of three sets of randomly selected units when selecting a small subset of units. error bars represent 95% confidence intervals calculated across models. see the other conditions). in contrast, responses from three sets of randomly sampled units show a different response profile (figure 4d ), demonstrating that the language response profile is not ubiquitously present throughout the llms. quantitative alignment between the language network in llms and brains. beyond qualitative alignment between llm language units and brains, we investigate the quantitative alignment to brain data. following standard practice in measuring brain alignment, we train a ridge regression to predict brain activity from model representations, using the same input stimuli presented to human participants in neuroimaging studies @xcite . we then measure the pearson correlation between the predicted brain activations and the actual brain activations of human participants on a held-out set. this process is repeated over 10 cross-validation splits, and we report the average (mean) pearson correlation as our final result which we here refer to as brain-score @xcite . figure 5 shows the average raw correlation when using {0.03125, 0.0625, 0.125, 0.25, 0.5, 1}% of model units to predict brain activity for two neural datasets @xcite . this analysis is repeated for the most language selective units, and for three sets of randomly sampled units for each of the 10 models. see appendix e for more statistical tests and appendix f for more dataset details. ## localizing the multiple demand & theory of mind networks the results so far suggest that the functional localization methods used in neuroscience to identify the brain's language network also applies effectively to llms. this raises a natural question: can we use the same localizers designed to identify other brain networks, such as the theory of mind network or the multiple demand network, to discover analogous networks in llms? ## discussion specialized llm language units. our findings provide compelling evidence that specialized language units emerge within llms. it is particularly surprising how effectively we can identify these units with the same limited set of localization stimuli employed in neuroscience, and that they prove to be causally relevant for language tasks. while we observed consistent results across all 10 models we tested, it remains an open question whether this specialization is universal across all llms and under which conditions this specialization does or does not emerge. for instance, does the nature of the training data or the specific training objective influence the emergence of these specialized units? moreover, the role of non-language-selective units remains unclear. it is possible they contribute to other specialized systems. while our experiments with the multiple demand and theory of mind selective units yielded some promising results, the variability across models suggests that these systems may either emerge more sparsely or be more complex or challenging to identify. consistency with the brain's language network. our paper adds to the growing body of research that uses neuroscience tools to interpret machine learning models @xcite @xcite . specifically, our work takes a step towards analyzing llm units that are causally useful within a given system, providing a more stringent notion of functional correspondence @xcite @xcite . the consistency between the causally important language units in llms and the human brain may suggest that computations, beyond representations, could be shared between these two systems. this prompts an intriguing question: do specialized subsystems, such as the language network, always emerge as a consequence of optimizing for next-word prediction, and is such a simple objective the driver of specialization in the brain? exploring this connection further could shed light on how cognitive processes evolve from such pre-10894 dictive mechanisms. related work previous work has identified a core language system within llms @xcite , but their approach requires finetuning the model on a next-token prediction task to locate parameters that exhibit minimal variation during finetuning. in contrast, our method bypasses additional training and leverages established research from language neuroscience. concurrently, @xcite have shown that llms exhibit brainlike functional organization by using regressors to predict brain activity based on artificial neuron responses, and thereby mapping llm representations onto the brain. however, their method is computationally intensive and lacks the cognitive neuroscience grounding that underpins our approach. other efforts have focused on identifying subnetworks that encode factual knowledge @xcite @xcite and task-specific skill neurons @xcite . ## conclusion in this paper, we explored whether functional specialization observed in the human brain can be identified in llms. drawing inspiration from neuroscience, we applied the same localizers used in human neuroscience, to uncover languageselective units within llms, showing that a small subset of these units are crucial for language modeling. our lesion studies revealed that ablating even a fraction of these units leads to significant drops in language performance across multiple benchmarks, while randomly sampled nonlanguage units had no comparable effect. al-though we successfully identified a language network analog in all models studied, we found mixed results when applying similar localization techniques to theory of mind and multiple demand networks, suggesting that not all cognitive functions neatly map onto current llms. these findings provide new insights into the internal structure of llms and open up avenues for further exploration of parallels between artificial systems and the human brain.