Lower Perplexity is Not Always Human-Like (2021)

Abstract: In computational psycholinguistics, various language models have been evaluated against human reading behavior (e.g., eye movement) to build human-like computational models. However, most previous efforts have focused almost exclusively on English, despite the recent trend towards linguistic universal within the general community. In order to fill the gap, this paper investigates whether the established results in computational psycholinguistics can be generalized across languages. Specifically, we re-examine an established generalization — the lower perplexity a language model has, the more human-like the language model is — in Japanese with typologically different structures from English. Our experiments demonstrate that this established generalization exhibits a surprising lack of universality; namely, lower perplexity is not always human-like. Moreover, this discrepancy between English and Japanese is further explored from the perspective of (non-)uniform information density. Overall, our results suggest that a cross-lingual evaluation will be necessary to construct human-like computational models.

## introduction it is well known that the probability of a word in context (i.e., surprisal) impacts its processing difficulty in incremental human language comprehension @xcite @xcite . building on this basis, researchers have compared a variety of language models (lms) in terms of how well their surprisal correlates with human reading behavior @xcite @xcite @xcite @xcite . such investigations could provide insights into the development of a general computational model of human language processing. for example, recent studies reported that lms with better performance for next-word prediction could also better predict the human reading behavior (i.e. more humanlike) @xcite @xcite . in this paper, we re-examine whether the recent findings on human-like computational models can be generalized across languages. despite the community's ongoing search for a languageindependent model @xcite , existing studies have focused almost exclusively on the english language. having said that, broad-coverage crosslinguistic evaluation of the existing reports is prohibitively difficult. in fact, data on human reading behavior (e.g., eye movement) is available only in limited languages. as an initial foray, this study focuses on the japanese language as a representative of languages that have typologically different characteristics from the english language. if the observation is different between english and japanese, the current findings on english data might lack a universality across languages. we specifically revisit the recent report-the lower perplexity a lm has, the more human-like the lm is-in the english and japanese languages @xcite @xcite . in addition to the importance of cross-linguistic evaluation, the report itself is worth investigating. recent studies in the machine learning field have reported that more parameters, training data, and computation cost can result in better ppl @xcite . our investigation has implications for whether a human-like model might exist beyond such improvements. more concretely, over three dozens of lms were trained for each language, with variants in their architecture, training data size, and the number of parameter updates. then, the surprisals computed by yononakaniwa (in the world) samazamana (all kinds of) hitoga (people) irutoiu (there are) kotoga (that) yoku (well) wakatta (understood) ## methods this section describes the settings of lms, eye movement data, and evaluation metrics. ## experiments the relationship between ppl and psychometric predictive power is investigated. furthermore, the relationship is analyzed with respect to the training configures of lms (e.g., the number of parameter updates). then, we discuss the results from the perspective of the uniformity of information density. bccewj-eyetrack (japanese) dundee corpus (english) psychometric predictive power psychometric predictive power figure 3: separate effect of model architecture, training data size, and the number of parameter updates for lms' psychometric predictive power in each language. each point corresponds to each lm. the box shows the quartiles of the data. the whiskers show 1.5 times interquartile range. lm (higher is better). the x-axis is ppl on a log scale (lower is better). dundee corpus: first, the results of the data from the dundee corpus show a clear relationship between ppl and psychometric predictive power; namely, lower ppl corresponds to more psychometric predictive power, as reported by prior studies @xcite ). spearman's rank correlation coefficient between the two metrics was -0.87. bccwj-eyetrack: by contrast, in bccwj-eyetrack, there was no clear, consistent trend between the ppl and psychometric predictive power. while lms with ppl over 400 show the correlation between ppl and psychometric predictive power (-0.68 with spearman's ρ), there is a positive correlation (0.53 with spearman's ρ) for lms with ppl below 400. the positive correlation means that the more accurately the lms can predict the upcoming word, the worse the psychometric predictive power of the lms is. these results demonstrate the non-universality of the recent report across languages; lower perplexity is not always human-like. the lstm lm trained using the md dataset with 1k updates achieved the best psychometric predictive power. notably, surprisal was effective for gaze duration modeling in all the japanese lms. ∆loglik scores were significantly higher than zero with the chi-square test (@xmath0.05). ## probing nonuniform information density of japanese lms this study hypothesized that tuning to the lm objective (i.e., uniform goal) obscures the nonuniform trend observed in the reading behavior of japanese subjects. we investigated whether the nonuniformity of the processing difficulty observed in human reading time is mirrored by lm surprisals. as @xcite reported, verbal and modifier segments have a shorter gaze duration than the other segments in japanese sentences. an analysis was conducted on how strongly the japanese lm's surprisals on segments are influenced by their syntactic category. this influence can be evaluated by examining how effectively syntactic category factors can model lm surprisals. in this experiment, surprisal was regarded as "simulated gaze duration" from an "lm subject," and the importance of syntactic category information for modeling the simulated gaze duration (simulated gd) was evaluated. to inspect the effect of the syntactic category information for modeling the simulated gaze duration, the follow-ing regression model 9 was used, including a factor defining which syntactic category the segment falls into (syn category): from this regression model, a log-likelihood score for the simulated gaze duration was obtained. to evaluate the separate effect of syn category, ∆loglik between eq. 4 and a baseline model was calculated. the baseline model was simulated gd ∼ sentn + tokenn + freq * length. the ∆loglik is denoted as "effect of syntactic category." a lower score means that the lm lacked the property of varying processing difficulty with respect to the syntactic category. results: the results are shown in figure 5 . first, the higher psychometric predictive power the lms exhibit, the greater the effect of syntactic category on surprisals (left part in figure 5 ). this means that, depending on the syntactic category of the segment they processed, lms with high psychometric predictive power computed surprisals with a more nonuniform trend. the right part of figure 5 shows that, as ppl decreases below a certain value (ppl ∼ 400), the japanese lms compute surprisals that obscure the nonuniform trends with 9 sentn and tokenn denote the sentence position and the segment position in a sentence (see appendix b). note that the tokenn and syntactic category exhibit low correlation (0.02 with pearson's r). notably, we also observed that not all the types of linguistic nonuniformity were obscured in surprisals computed by the lms with low ppl. for example, appendix e shows that lms with lower ppl compute surprisals that better correlates with a particular syntactic factor although that factor is a less dominant trend in human reading behavior than the syntactic category (appendix d). ## limitations and future works to test the universality of the recent findings in computational psycholinguistics across languages, the initial focus is on english and japanese as a pair of languages with different linguistic properties. although the discrepancy of the results in the two languages is discussed from the viewpoint of the uid hypothesis, the two languages are also different in various ways, such as writing systems, agglutinative property, case marking, sentence structure, and pro-drop nature. to identify the difference that relates to the human-like behaviors of lms, experiments that include additional languages should be conducted in the future. in addition, the corpus size of the bccwj-eyetrack data is smaller than the dundee corpus. while the reading time data in the bccwj-eyetrack was collected from various human subjects, the number of the independent segments was limited (1,643 segments, 218 sentences). thus, whether the trends reported in this study generalize to more diverse japanese texts should be explored in future work. it is hoped that this study motivates the creation of a large-scale corpus of human reading behaviors in diverse languages. ## conclusion this study has investigated whether the recent reports on the psychometric predictive power of lms can be generalized across languages. our initial investigation has re-examined the recent report-the lower ppl a lm has, the more human-like the lm is-using japanese eye movement data. our experiments have demonstrated a surprising lack of universality of this report; lower perplexity is not always human-like. this discrepancy of the results between the languages reinforces the need for the cross-lingual evaluation of the psychometric predictive power of lms. the discussion considers potential factors that make the observation different across languages from the viewpoint of the uniform information density hypothesis. we believe that this is an important first step for seeking a language-agnostic model of human sentence processing. hopefully, this study encourages researchers to further investigate the universality of human language processing across languages.