A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing (2023)

Abstract: Two of the central factors believed to underpin human sentence processing difficulty are expectations and retrieval from working memory. A recent attempt to create a unified cognitive model integrating these two factors have relied on the parallels between the self-attention mechanism of transformer language models and cue-based retrieval theories of working memory in human sentence processing (Ryu and Lewis 2021). While the authors show that attention patterns in specialized attention heads of GPT-2 are consistent with a key prediction of cue-based retrieval models, similarity-based interference effects, their method requires the identification of syntactically specialized attention heads, and makes an cognitively implausible implicit assumption that hundreds of memory retrieval operations take place in parallel. In the present work, we develop a recurrent neural language model with a single self-attention head, which more closely parallels the memory system assumed by cognitive theories. We show that our model’s single attention head can capture semantic and syntactic interference effects observed in human experiments.

## introduction theories of human sentence processing can be divided into two broad categories. expectation-based theories like surprisal theory @xcite @xcite ascribe processing difficulty to a word's predictability. memory-based theories, such as cue-based retrieval @xcite @xcite @xcite , account for processing difficulty incurred due to limitations on working memory encoding and retrieval. integrating these approaches has become a major goal of sentence processing research @xcite @xcite @xcite . surprisal theory argues that the processing difficulty associated with a word is linearly related to its negative log probability, or surprisal, in context. surprisal theory is agnostic by design to the precise mechanisms underlying sentence processing; models with drastically different representational assumptions can be used to obtain surprisal estimates @xcite . the advent of neural network language models (nlms), in particular, has broadened surprisal theory's empirical coverage to a wide range of phenomena @xcite @xcite . however, nlms' opaque internal representations make it difficult to relate them to theories of how language is represented and retrieved from memory by humans (though see @xcite . by contrast, cue-based retrieval theories make explicit claims about how working memory is structured and accessed during language comprehension. according to these theories, humans have a narrow attentional focus, and therefore must retrieve items from working memory to resolve longdistance linguistic dependencies. items in working memory are accessed by matching a set of featural retrieval cues provided by the current word in parallel against the features of all items in working memory. the item which best matches the cue features is retrieved and used to parse the current word. the speed and accuracy of retrieval is affected by the number of items in memory that match the retrieval cue features. the effect of similarity of the target and distractor items on retrieval difficulty, referred to as similarity-based interference, is a key prediction of cue-based retrieval models, and has been used to explain various phenomena in the sentence processing literature @xcite . despite their empirical successes, implemented models of cue-based retrieval traditionally employ only a small set of hand-picked linguistic features, such as [+/-subject] or [+/-singular]. this limits model coverage, increases modeler degrees of freedom and decreases theoretical parsimony @xcite . by contrast, nlms learn all of their linguistic features implicitly as they optimize a word prediction training objective. leveraging this ability of nlms, ryu and @xcite propose a synthesis of surprisal and cuebased retrieval theories. they note the parallels between cue-based retrieval and neural self-attention in transformer nlms @xcite . to generate a contextualized representation of the current word, self-attention heads score the representations of all context words against a query generated at the current word, and sum the context word representations weighted by these attention scores. transformer nlms, such as gpt-2 @xcite , use many attention heads simultaneously, arranged into multiple layers. ryu and lewis show that attention patterns in a syntactically-specialized head in gpt-2 are consistent with similarity-based interference explanations of sentence processing difficulty. crucially, in their model, all of the features used to compute attention scores were implicitly learned through the model's next word prediction training objective; no linguistic features need to be hand-picked. despite some parallels between transformers like gpt-2 and cue-based retrieval models, however, there are crucial differences in their working memory limitations. the smallest gpt-2 model has 144 attention heads. under the view of selfattention as cue-based retrieval, this would mean that 144 distinct retrievals from working memory take place at each timestep. by contrast, cue-based retrieval theories argue that working memory is highly constrained; a very limited number of items can be retrieved from memory within the time it takes to process a single word. if an nlm is to truly integrate expectation and memory-based theories, then it must be consistent with assumptions about human working memory. in this work, we present a recurrent nlm with memory limitations that are more closely aligned with cue-based retrieval theories. figure 1 shows how we conceptualize memory retrieval in our model. we use self-attention to implement cuebased memory retrieval, but employ only a single attention head, limiting the model to a single retrieval operation per timestep. in contrast to prior approaches, we do not need to rely on post-hoc identification of syntactically specialized attention heads. in our first experiment, we confirm that our model's single attention head learns to track subject-verb dependencies without direct supervision. in our main experiment, we show that measures of memory retrieval difficulty derived from our model's attention mechanism capture both semantic and syntactic retrieval interference effects from a human experiment @xcite , without the need to handpick any linguistic features as in existing cue-based retrieval theories. foot_0 2 background: attraction and similarity-based interference ## experiments we first examined whether the attention mechanism in trained cbr-rnn models is sensitive to longdistance subject-verb dependencies, even though it does not receive direct supervision in training as to the words that should be attended to. then, in our main experiment, we evaluated whether language model surprisal and a measure of similarity-based interference from our model's attention mechanism capture semantic and agreement attraction effects using experimental stimuli from laurinavichyute and von der malsburg (2022). ## conclusion in this work, we proposed a neural model which incorporates memory constraints from cue-based retrieval theories. we showed that the model's attention mechanism tracks subject-verb dependencies over long distances and with intervening distractor nouns. finally, we evaluated whether the model's attention component tracks similarity-based interference effects found in human attraction experiments. we showed that both model surprisal estimates and relative attention from our model's selfattention mechanism both predicted the presence of semantic and agreement attraction, without the need to explicitly stipulate any linguistic features in the model. however, relative attention generally under-predicted plausibility ratings and the relative size of the semantic attraction effect. taken together, we showed that mechanistic explanations of processing difficulty rooted in memory retrieval constraints can emerge as a by-product of minimizing the surprisal of upcoming words. high performance computing resources, services, and staff expertise. we would also like to thank suhas arehalli and members of the nyu computation and psycholinguistics lab for their insightful feedback and thought-provoking discussion.