Context Limitations Make Neural Language Models More Human-Like (2022)

Abstract: Language models (LMs) have been used in cognitive modeling as well as engineering studies—they compute information-theoretic complexity metrics that simulate humans’ cognitive load during reading.This study highlights a limitation of modern neural LMs as the model of choice for this purpose: there is a discrepancy between their context access capacities and that of humans.Our results showed that constraining the LMs’ context access improved their simulation of human reading behavior.We also showed that LM-human gaps in context access were associated with specific syntactic constructions; incorporating syntactic biases into LMs’ context access might enhance their cognitive plausibility.

## introduction in computational psycholinguistics, human reading behavior has been compared with various complexity metrics to understand human sentence processing @xcite . having historically started from simple measures such as word length, surprisal (log p(word|context)) computed by language models (lms) has become a common choice @xcite . on top of this, the next question arises-which model implementation and/or algorithm can compute surprisal that successfully simulates human behavior? in this line of research, modern neural lms such as transformer @xcite have been analyzed with respect to their cognitive plausibility @xcite @xcite . despite their use in cognitive modeling, such modern lm architectures (e.g., self-attention) are, arguably, an unnatural choice when it comes to human cognitive constraints; modern lm architectures assume powerful, parallel access to a vast 1 our codes are available at https://github. com/kuribayashi4/context_limitation_cognitive_ modeling number of context tokens, while humans might have limited and selective context access @xcite @xcite . searching for a computational model that better simulates human sentence processing than previously examined ones, we hypothesized that introducing such context limitations can improve lms' estimation of human cognitive load. specifically, as a starting point, we applied an n-gram-ification trick to neural lms mimicking loading for long context access (locality effects) and compared their surprisal with human reading behavior data. despite the simple context limitation design, our experiments with 280 settings (40 lm settings×7 noise patterns) showed that the advantage of a shorter context was consistent among neural model architecture data size training steps tokenization syntactic supervision context length goodkind and bicknell (2018); aurnhammer and frank (2019); wilcox et al. (2020); hao et al. (2020); merkx and frank (2021); oh et al. (2021); kuribayashi et al. (2021) wilcox et al. (2020); kuribayashi et al. (2021) kuribayashi et al. (2021) wilcox et al. (2020); oh et al. (2021) hale et al. (2018); yoshida et al. lms and typologically different languages (figure 1 ). this showed that constraining the modern lm's context access is key to increasing their similarity to the model of human reading. furthermore, expecting that humans' context limitations might be more complex than simple distance-based erasure, we conducted exploratory analysis of in which constructions longer/shorter contexts were beneficial. we found that the context limitation (dis)advantages were allocated in specific syntactic constructions, suggesting that, to build more cognitively plausible lms, adding syntactic biases in their context access could be beneficial. from a psycholinguistic view, our results empirically highlight the memory account of human sentence processing during naturalistic reading @xcite . ## methods we investigate how human-like neural lms become with more or less context at their input. specifically, we measured the psychometric predictive power (ppp) of lossy-context lm surprisal for gaze duration modeling. in the following sections, we describe each measure in detail. ## language models we used two types of neural lms for lossy-context (n-gram) surprisal computation: (i) wiki-lms and (ii) pretrained openai gpt-2s @xcite . their hyperparameters are shown in appendix d. notably, using neural lms makes the comparison of long-context and short-context lms computationally tractable. foot_1 ## experiments our experiments demonstrate how limiting context access improved the ppp in lms, i.e., their surprisal becomes a more effective predictor for human gaze duration (section 5.1). as described in section 3.2, we applied distance-based noise to the input (i.e., computing n-gram surprisal). a potential training-inference mismatch bias is handled (section 5.2). furthermore, we explored the connection to existing studies (section 7.2). settings. we measured the ppp of 40 lms ({3 wiki-lms}×{3 seeds}×{2 languages}×{2 training settings}+{4 openai gpt-2s}) with seven noise patterns. specifically, we explored the {2, 3, 5, 7, 10, 20}-grams and full settings for each lm, where full refers to using the entire sentence (w 0 to w i-1 ) as context. note that only intrasentential context is used as the main focus of this study is sentence-level syntactic processing. ## analyses in our experiments, we merely deleted distant contexts regardless of linguistic factors. however, this design is somewhat counter-intuitive in the sense that humans are assumed to completely forget the distant context during reading. to gain insights into a more sophisticated noise design to fill the lm-human gap, we observed in which constructions longer/shorter contexts improved simulation of human gaze duration. ## conclusions there has been little investigation of the cognitive plausibility of context-limited modern lms. our experiments using the input-controlled neural lms have shown that short context lms simulate human reading behavior surprisingly well, emphasizing the lm-human gap in context access. further analysis has shown that the gap could be associated with specific syntactic constructions; injecting syntactic bias into lms' context access could be one way to make lms more human-like. this study has also asserted that the use of a modern lm popular in nlp as-is is not always a natural choice in cognitive modeling.