  The activations of language transformers like GPT2 have been shown to
linearly map onto brain activity during speech comprehension. However, the
nature of these activations remains largely unknown and presumably conflate
distinct linguistic classes. Here, we propose a taxonomy to factorize the
high-dimensional activations of language models into four combinatorial
classes: lexical, compositional, syntactic, and semantic representations. We
then introduce a statistical method to decompose, through the lens of GPT2's
activations, the brain activity of 345 subjects recorded with functional
magnetic resonance imaging (fMRI) during the listening of ~4.6 hours of
narrated text. The results highlight two findings. First, compositional
representations recruit a more widespread cortical network than lexical ones,
and encompass the bilateral temporal, parietal and prefrontal cortices. Second,
contrary to previous claims, syntax and semantics are not associated with
separated modules, but, instead, appear to share a common and distributed
neural substrate. Overall, this study introduces a general framework to isolate
the distributed representations of linguistic constructs generated in
naturalistic settings.
