Hierarchical syntactic structure in human-like language models (2024)

Abstract: Language models (LMs) are a meeting point for cognitive modeling and computational linguistics. How should they be designed to serve as adequate cognitive models? To address this question, this study contrasts two Transformer-based LMs that share the same architecture. Only one of them analyzes sentences in terms of explicit hierarchical structure. Evaluating the two LMs against fMRI time series via the surprisal complexity metric, the results implicate the superior temporal gyrus. These findings underline the need for hierarchical sentence structures in word-by-word models of human language comprehension.

## introduction interest in language models (lms) has exploded due to their recent success on language-related tasks @xcite , with many commentators speculating about their implications as models of human language processing (see @xcite ii, for a review). the methodological utility of natural language processing tools for isolating language-processing functions in the brain is by now well-established @xcite @xcite @xcite ; however, controversy persists regarding the role of hierarchical structure as useful or not in characterizing human language comprehension (e.g., @xcite , yielding two related questions. 1. is hierarchical structure part of the best description of human language comprehension? 2. if so, what brain regions subserve this aspect of processing? this study investigates these questions by comparing two language models with the same underlying architecture. one is constrained via a special attention mask that captures hierarchical structure in the form of syntactic constituency, while the other lacks this attention mask, capturing only wordlevel information. the hierarchy-biased model is a transformer grammar (tg; @xcite , which differs only from the unconstrained model, transformer-xl (txl; @xcite , in the presence of this attention mask. we pair these language models with surprisal, a word-by-word information-theoretic complexity metric (see @xcite , for a review) to derive predictions about neuroimaging data. surprisal from the hierarchy-biased tg compares against surprisal from the unconstrained txl in the task of predicting fmri data @xcite . this sets up a clean contrast between hierarchical and non-hierarchical conceptions of language comprehension. the results, reported in section 6, support the role of hierarchical structure in language comprehension. surprisal values derived from a transformer grammar predict fmri timecourses in bilateral superior temporal gyrus (stg) better than those from txl. this supports the view that the stg is sensitive to hierarchical sentence structure @xcite . ## phrase structure the penn treebank operationalizes one notion of hierarchical structure @xcite . the present study uses these trees, exemplified in figure 1 . the syntactic analyses that they express date back to @xcite and can be motivated by considerations such as substitution, compositionality and structuredependence of transformational rules which are reviewed in introductory linguistics textbooks (e.g. @xcite . for a broad, comparative discussion of hierarchical structure in language, see @xcite . ## transformer grammar transformer grammars @xcite model the joint-probability of a surface string x and its corresponding phrase structure tree y, p(x, y). they incorporate an inductive bias toward hierarchical syntax via special attention masks. these attention masks mark the only difference between tg and a general transformer-xl @xcite . tgs apply the idea of parsing as language modeling @xcite @xcite by assigning probability to labelled, bracketed strings. they innovate on that idea by restricting -via the additional attention mask -the information used in label assignment. this information is restricted to prior composed phrases and the direct subconstituents of the current phrase being composed. these restrictions result in stack representations that correspond to levels of a syntactic derivation (for more details on tg's recursive syntactic composition see @xcite . ## previous work investigating hierarchy using computational modeling and neuroimaging data this work builds on research that compares wordby-word difficulty predictions against neuroimaging data. previous work of this type has found support for hierarchical structure @xcite @xcite @xcite others, following in the tradition of elman (e.g., 1990, see also @xcite , have questioned the need for hierarchical structure. proponents of this view point to the successes of lms that rely just on overt word sequences in encoding (or decoding) human brain responses to language (e.g., @xcite @xcite ; see @xcite for a review). the most extreme form of this view holds that word-prediction alone suffices to explain human language processing @xcite . the present study addresses this debate regarding the role of hierarchy in language comprehension by comparing two language models with the same underlying architecture, the only difference being that hierarchical structure is explicitly present (vis-a-vis the additional attention mask) in one (the tg) and not in the other (the txl). ## results the addition of surprisal derived from the hierarchy-biased tg model performed above-andbeyond the addition of surprisal derived from the unconstrained txl model in goodness-of-fit (r 2 ) to the measured bold signal in bilateral stg (fig. 3 ; ## discussion the findings support the role of stg in hierarchically-sensitive sentence processing @xcite . notably, the results for surprisal in stg are largely localized to auditory cortex (see also @xcite . these results suggest, in line with the sensory hypothesis @xcite , that hierarchical structure from earlier in the sentence can impact low-level sensory processing. prior investigation into early (< 150 ms) processing using meg has found that auditory cortex is sensitive to phrase structure @xcite . this early sensitivity to hierarchical structure indicates that previously encountered structure may modulate sensory processing of subsequent words in a top-down manner. employing a precise regression analysis and holding the architecture of lms constant, the current study offers novel evidence in support of the sensory hypothesis and the early influence of hierarchical structure in language comprehension. one region that has been largely implicated in predictive processing such as the type modeled here (e.g., @xcite @xcite is the left inferior frontal gyrus (lifg). the present study does not implicate lifg. it is possible that this null result could be due to the fact that the level of prediction and prediction violation here is too modest to invoke the lifg, which seems more associated with processing particularly complex stimuli. the success of modern lms in natural language processing tasks has revived hope (see §4) that hierarchical structure could be left out of an adequate cognitive model. the results reported here suggest contrariwise. this echos @xcite , who find that lms strongly under-predict human reading time on syntactically challenging constructions, @xcite who differentiate lm layers that better-predict successor words from layers that better-predict fmri data, and @xcite , who find that unbiased lms fail to generalize structurally-dependent constructions in a human-like way. with antonello and huth, we acknowledge that unconstrained lms learn something about syntax. but it is not enough; in the context of cognitive modeling, additional bias towards hierarchical structure seems to be needed @xcite . ## conclusion hierarchical structure remains a key part of the best characterization of human language comprehension. this conclusion rests upon the increase in bold r 2 from the addition of tg-derived surprisal compared to the addition of txl-derived surprisal. this obtains in a well-known temporal node of the language network and shores up the view of the language-processing brain as a system that performs hierarchical combinatorics. the results here also support recent arguments against unbiased lms as cognitive models of human language.