Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars (2021)

Abstract: In computational linguistics, it has been shown that hierarchical structures make language models (LMs) more human-like. However, the previous literature has been agnostic about a parsing strategy of the hierarchical models. In this paper, we investigated whether hierarchical structures make LMs more human-like, and if so, which parsing strategy is most cognitively plausible. In order to address this question, we evaluated three LMs against human reading times in Japanese with head-final left-branching structures: Long Short-Term Memory (LSTM) as a sequential model and Recurrent Neural Network Grammars (RNNGs) with top-down and left-corner parsing strategies as hierarchical models. Our computational modeling demonstrated that left-corner RNNGs outperformed top-down RNNGs and LSTM, suggesting that hierarchical and left-corner architectures are more cognitively plausible than top-down or sequential architectures. In addition, the relationships between the cognitive plausibility and (i) perplexity, (ii) parsing, and (iii) beam size will also be discussed.

## introduction it has been debated in computational linguistics whether language models (lms) become more human-like by explicitly modeling hierarchical structures of natural languages. previous work has revealed that while sequential models such as recurrent neural networks (rnns) can successfully predict human reading times @xcite , there is an advantage in explicitly modeling hierarchical structures @xcite . more recently, rnns that explicitly model hierarchical structures, namely recurrent neural network grammars @xcite , have attracted considerable attention, effectively capturing grammatical dependencies (e.g., subject-verb agreement) much better than rnns in targeted syntactic evaluations @xcite . in addition, @xcite showed that rnngs can successfully predict human brain activities, and recommended rnngs as "a mechanistic model of the syntactic processing that occurs during normal human sentence processing." however, this debate has focused almost exclusively on the dichotomy between the hierarchical and sequential models, without reference to the parsing strategies among the hierarchical models. specifically, although @xcite and @xcite adopted the vanilla rnng with a top-down parsing strategy for english with headinitial right-branching structures, @xcite and @xcite suggested that the top-down parsing strategy is not optimal for headfinal left-branching structures, and alternatively proposed the left-corner parsing strategy as more human-like parsing strategy. in this paper, we investigate whether hierarchical structures make lms more human-like, and if so, which parsing strategy is most cognitively plausible. in order to address this question, we evaluate three lms against human reading times in japanese with head-final left-branching structures: long short-term memory (lstm) as a sequential model and recurrent neural network grammars (rnngs) with top-down and left-corner parsing strategies as hierarchical models. ## linking hypothesis it is well established in psycholinguistics that humans predict next segments during sentence processing, and the less predictable the segment is, the more surprising that segment is. surprisal theory @xcite quantifies this predictability of the segment as -log p(segment|context), an information-theoretic complexity metric known as surprisal. in line with the previous literature (e.g., @xcite , we employed this metric to logarithmically link probabilities estimated from lms with cognitive efforts measured from humans. intuitively, the cognitively plausible lms will compute surprisals with similar trends as human cognitive efforts. computational models of human sentence processing have been explored by comparing surprisals from various lms with reading times (e.g., @xcite and brain activities (e.g., @xcite . ## results and discussion the result of our computational modeling is summarized in figure 1 : psychometric predictive power (the vertical axis) is plotted against perplexity (the horizontal axis). 7 in this section, we first analyze psychometric predictive power itself, and then discuss its relationships with (i) perplexity, (ii) parsing, and (iii) beam size. ## limitations and future work interestingly, @xcite demonstrated that top-down rnngs underperformed lstms in predicting human reading times in english, which appears to be contradictory to our results in japanese. we would like to suggest that this discrepancy can be attributed to the difference in the languages tested in these papers. in fact, @xcite have shown that several established results in english cannot be straightfor-wardly generalized to japanese. in addition, @xcite found that ngram language models outperformed various neural language models, while @xcite observed that transformers @xcite outperformed lstms in modeling self-paced reading times and n400 brain activities, but not in predicting eye-tracking reading times. in order to address these limitations, we plan to conduct detailed comparisons between english (dundee corpus, @xcite and japanese (bccwj-eyetrack, @xcite with rnngs, incorporating n-gram language models and transformers as baselines in future work. ## conclusion in this paper, we investigated whether hierarchical structures make lms more human-like, and if so, which parsing strategy is most cognitively plausible. our computational modeling demonstrated that left-corner rnngs outperformed top-down rnngs and lstm, suggesting that hierarchical and left corner architectures are more cognitively plausible than top-down or sequential architectures. moreover, lower perplexities and higher parsing accuracies of the hierarchical models were strongly correlated with the higher psychometric predictive power, but the correlation did not hold for the sequential model. in addition, left-corner rnngs may be more human-like than top-down rnngs in that they can infer the correct tree structure with a small beam size comparable to humans.