Individual corpora predict fast memory retrieval during reading (2020)

Abstract: The corpus, from which a predictive language model is trained, can be considered the experience of a semantic system. We recorded everyday reading of two participants for two months on a tablet, generating individual corpus samples of 300/500K tokens. Then we trained word2vec models from individual corpora and a 70 million-sentence newspaper corpus to obtain individual and norm-based long-term memory structure. To test whether individual corpora can make better predictions for a cognitive task of long-term memory retrieval, we generated stimulus materials consisting of 134 sentences with uncorrelated individual and norm-based word probabilities. For the subsequent eye tracking study 1-2 months later, our regression analyses revealed that individual, but not norm-corpus-based word probabilities can account for first-fixation duration and first-pass gaze duration. Word length additionally affected gaze duration and total viewing duration. The results suggest that corpora representative for an individual’s long-term memory structure can better explain reading performance than a norm corpus, and that recently acquired information is lexically accessed rapidly.

## introduction there are three basic stages of memory (e.g. @xcite . all memories start with experience, which is reflected by text corpora (e.g. @xcite . the training of a language model then reflects the process of memory consolidation. the final stage is memory retrieval, which can be examined in psycholinguistic experiments. in this paper, we measure the correlation of computational language modelling and cognitive performance. we collected individual corpora from two participants reading on a tablet for two months and compared them to an extensive corpus mainly consisting of online newspapers @xcite . to consolidate differential knowledge structures in long-term memory, word2vec models were trained from these corpora. for stimulus selection, we relied on these three language models to compute word probabilities and sentence perplexity scores for 45k sentences of a wikipedia dump. perplexity rank differences were used to select sentences with uncorrelated word probabilities for the three language models, allowing to estimate the independent contribution of the word probabilities in multiple regression analyses. the resulting 134 stimulus sentences were read by the participants in an eye tracking experiment. in the multiple regression analyses, we used these predictors to account for the durations of the first fixation on the words. we also predicted gaze durations, in which the duration of further fixations during first-pass reading are added. when the eye revisits a word after first-pass reading has been finished, the durations of further fixations are added into the total viewing duration (see figure 1 for an overview of the present study). the underlying hypotheses of our research are that semantic expectancy has a topdown effect on word saliency at the visual level @xcite , and words appearing in more salient contexts, are processed quicker by human subjects. therefore, language models on individual reading corpora, realized e.g. by word2vec, should predict the processing speed. 2 learning history and memory consolidation 2.1 norm corpora as a representative sample of human experience? when selecting a corpus as a sample of the learning experience of human participants or language models, the question arises which corpus is most representative for which person. for instance, the knowledge of young adults is better characterized by corpora consisting of books written for younger adults, while older adults are more experienced -when therefore searching for corpora that account for their performance best, more diverse fictional and literature books are chosen @xcite . @xcite proposed that corpus representativeness should be measured by the pearson correlation of corpus-derived computational measures and an external measure of human performance. the knowledge of an average reader may be well represented by balanced corpora containing all sorts of content, such as wikipedia. however, a previous study revealed that a newspaper corpus often provides higher correlations than wikipedia when accounting for human cloze completion probabilities, as well as eye tracking or brain-electric data @xcite . this result pattern could on the one hand be explained by the fact that typical german readers very frequently visit online newspapers. on the other hand, the encyclopedic nature of wikipedia with one article per topic does not reflect the frequencies of exposure. therefore, a newspaper corpus may be more representative. in his seminal work, @xcite avoided the possibility to observe the average language input of test persons, noting that it would be effortful and unpractical to collect corpora from the language of participants. in the present work, we directly address this problem by collecting individual corpora from two test participants, which is made possible by technological advances and cheaper hardware, i.e. tablets with cameras for eye-tracking. when choosing a corpus, a key issue is its size. in general, a larger corpus may provide better word frequency estimates that allow for better human performance predictions than a small corpus @xcite . for high frequency words, however, a corpus of 1 million words already allows for predictions comparable to larger corpora @xcite . for low frequency words, performance predictions improve for corpora up to 16 million words, while there is hardly any gain for corpora greater 30 million words. rather, extremely large corpora of more than a billion words may even decrease performance predictions. "for these sizes, it becomes more important to know where the words of the corpus came from" @xcite . movie subtitles represent spoken language, which humans typically encounter much more often than written language. in this case, the corpus is probably more representative, because it incorporates the more frequently produced and received language mode. therefore, it appears that the representativeness for the language input can outperform sheer size. whether such smaller corpora are also sufficient to characterize contextual word probabilities has also been examined by @xcite . they showed that size does not always trump representativeness in predicting semantic priming, i.e. the facilitation of word recognition as a consequence of processing a preceding prime word. in general, the everyday social information represented in subtitles corpora may be more accessible for human subjects. therefore, they tend to elicit effects in early eyemovement measures, particularly when combined with a language model that can generalize, such as neural network models, for instance @xcite . though personalized language signals are obviously used to optimize search engine performance, tempting us to purchase products by means of individualized advertisements, we are not aware of any scientific approach towards assessing corpus representativeness at the level of real individual participants. @xcite , however, showed that language models can well characterize fictional characters. he used sentiart to analyze harry potter books and found that theory-guided contextual properties of the characters can provide a face valid approach to personality. voldemort occurred in language contexts indicating emotional instability, thus he scored high in the pseudo-big-five personality trait of neuroticism. harry potter's personality, in contrast, can be most well characterized by the personality dimension of conscientiousness. both, harry and hermione score high on the personality trait of openness to experience and intellect. with such a face valid characterization of the personality of fictional characters, we think that it is an ethical necessity to stimulate a scientific discussion about the potential of individual corpora, because they may tell us a lot about real persons. ## memory retrieval in eye-tracking analyses as has been summarized in figure 1 and already introduced above, there are a number of different eyemovement parameters that can be used to address early and later memory retrieval processes during sentence reading (e.g. @xcite . when the eyes land on a word within a sentence during left-to-right reading, they remain relatively still for a particular amount of time, generally referred to by the term fixation duration. the first fixation on a word duration (ffd) is generally assumed to reflect early orthographic and lexical processing @xcite ), but has also been shown to be sensitive for readily available predictive semantic (top-down) information for a given word (e.g. @xcite . the sum of all fixation durations before the eye leaves the word to the right is referred to by (first-pass) gaze duration (gd), which reflects later stages of word processing including lexical access. after leaving the word to the right, the eye may come back to the respective word and remain there for some time, which is further added into the total viewing duration (tvd). such late eye movement measures reflect the time needed to provide full semantic integration of a word into the current language context @xcite . word length, frequency and word predictability from sentence context are generally accepted by the eye-tracking community to represent the most influential psycholinguistic variables on eye-movements (e.g. @xcite . word length is particularly affecting medium to late cognitive processes, while word frequency seems to affect all eye-movement measures (e.g. @xcite . in psychology, word predictability from sentence context is typically estimated by cloze completion probabilities @xcite , which can be well approximated by language models @xcite . there are numerous studies examining the influence of predictability on eye movement measures, which found that predictability affects both early and late eye movement parameters: therefore, @xcite review suggested that cloze-completion-probability-based predictability is an all-in variable confounding all sorts of predictive processes. we believe that language models provide the opportunity to understand different types of "predictability", therefore allowing for a deeper understanding of how experience shapes memory and how memory acts on retrieval than current models of eye-movement control @xcite . while @xcite proposed to use single-predictor regressions to approach corpus representativeness, a typical analytic approach to eye movements are multiple regression analyses. in this case, the fixation durations are approximated by a function: f(x) = ∑ bn*xn + b + error (1) in formula 1, xn is the respective predictor variable such as length, frequency or predictability, and the free parameters are denoted by b. bn reflects the slope explained by the predictor variable n, while b is the intercept of the regression equation. error is minimized by ordinary least squares. in single-predictor analyses, correlation coefficients inform about the relative influence of a single variable. to see how much variance is explained by a single predictor, the correlation coefficient is often squared to give the amount of variance explained. though the typical variance explained by a single predictor can vary as a function of the variables included in the regression model, an r 2 = 0.0095 (@xmath0.097) for the frequency effect in gd is a good benchmark at this fixation-based level of analysis (e.g. @xcite , a critical factor influencing multiple regression analyses is the correlation of the predictor variables itself. if they surpass an @xmath1.3, multicollinearity starts to become problematic and the variance inflation factor reaches a first critical level of 1.09 (e.g. @xcite . therefore, we here relied on a sentenceperplexity-and word-probability-based stimulus selection procedure, to allow for an independent prediction of the major variables of interest, i.e. our word2vec-based word probabilities (wp) of the individual and the norm-based training corpora. ## discussion to examine the representativeness of different training corpora for the learning experience, we here collected corpora for two individuals. these individual corpora were compared against a norm-based corpus @xcite . we computed individual and norm-based semantic long-term memory structure based on word2vec models @xcite . then we computed wps for a sample of 45k sentences from wikipedia and selected 134 sentences providing no significant correlation between the norm-based and the individual wps to avoid multicollinearity in our multiple regression analyses on eye movement data. single-predictor and multiple regression analyses revealed that there are no significant effects of norm-based wp with any eye movement measure. one possible reason for these zero findings could be that we replaced words with a @xmath2, but not under the norm-based corpus. this might have increased the sensitivity for successful predictions of individual corpora. the stimulus selection procedure optimized the eye-tracking experiment for the representativeness for our participants' individual knowledge structure. this may compromise the representativeness for other types of knowledge. the individual wp, in contrast, revealed reliable single-predictor effects. these effects were largest in ffd data and decreased for later gd and tvd data. when comparing the 1.44% of explained variance of individual wp in the ffd data to the total variance of 1.77% explained in the multiple regression, this result pattern suggests that most of the variance was explained by individual wp. the slight increase in explained variance primarily results from norm-based wf, which marginally failed to reach significance in the multiple regression analysis on ffd data. in general, the amount of explained variance in these analyses are comparable to other multiple regression studies predicting each fixation duration, without aggregating across eye-movement data (e.g. @xcite . single and multiple regression analyses revealed that the effect of individual wp tends to become smaller, but apparent in gd data. this suggests that individual corpora are most suitable to predict early to mid-latency eye movement measures (cf. figure 1 ). in the multiple regression analysis of gd data, an additional effect of word length was observed, confirming the finding that length has a larger effect on such later eye movement parameters (e.g. @xcite , because multiple fixations are more likely in longer words. in tvd data, we observed no effect of individual wp, but a large effect of word length in the multiple regression analyses. the largest limitation of the present study is the low statistical power of eye tracking data from two participants only. as @xcite already noted, the collection of individual corpora is effortful, but we think that this effort was worthwhile, even when the present study relied on a limited amount of statistical power. we were positively surprised by the reliable effect of individual wp in early and mid-range eye movement parameters. nevertheless, the present work should be seen as a pilot study that will hopefully encourage further examinations of individual corpora. but there is also a second power issue that makes these results convincing. our norm-based training corpus was at least 140 times larger than the individual training corpora. therefore, we think that this is sound evidence that representativeness of a corpus for individual long-term memory structure can outperform size in predicting individual reading performance (e.g. @xcite . one reason for our conclusion that individual corpora may better predict eye movements lies in the time period, in which the text corpus reflecting human experience was acquired. the individual corpora were collected in a two-month time period that preceded the eye tracking study by about 1-2 months. therefore, the individual corpora may primarily reflect more recently acquired knowledge. @xcite proposed multiple buffer stores in their theory of long-term working memory (cf. @xcite . recently acquired knowledge is held in these buffers before it is integrated into long-term memory. therefore, participants can stop reading a text, and when carrying on reading later, only the first sentences are read slower as compared to continuous reading of these texts @xcite . our early eye-movement effects may be explained by the proposal that the recently acquired knowledge still resides in @xcite long-term buffer stores. while they argue that football knowledge can well predict comprehension of football-related texts, for instance, it is hard to answer the question of which knowledge has been acquired in which time period. first, individual corpora may help to free such studies from the investigation of one particular type of knowledge, because each individual corpus reflects the knowledge of this individual. second, individual corpora collected at different time periods may provide a novel approach to the question of how long information may persist in these knowledge buffers. our results suggest that information still residing in long-term memory buffers elicits faster and more efficient memory retrieval. there are several studies, in which participants are required to write diaries, which can be considered as extremely small individual corpora (e.g. @xcite . another example is the task to write emails to predict individual traits: for instance, @xcite found that participants with high extraversion tend to use "will not" for expressing negation, while participants with low extraversion tend to use "not really" (see @xcite , for a recent overview). while such studies focused on the language output, the present study provided two input variables. first, individual corpora allow to estimate individual experience. second, we selected the materials based on a language model to specifically capture the individual experience of the participants in the eye-tracking experiment. the language models can be considered an algorithmic approach to the neurocognitive system between the inputs and the output. much as differences of the cognitive architecture of the participants, there are "interindividual" differences between language models. for instance, n-gram models may reflect the capability of participants to remember particular words in context of specific other words, while neural network models more closely reflect the human capability to generalize from experiences (e.g. @xcite . thus, comparing such models with respect to the question of which model predicts which participant may provide information about the cognitive architecture of the respective participant. with respect to human intelligence testing, individual corpora should be a suitable approach to face @xcite challenge of "freeing adult tests from assumptions of uniform knowledge". for future work, we would like to proceed in three directions. first, we would like to improve the collection procedure: the corpora collected via screenshots and ocr contain a high number of artifacts stemming from non-textual material, as well as non-contiguous texts as a result from complex webpage layouts. second, we like to increase the number of participants in future studies. third, it would be interesting to compare our word2vec results with more recent contextual embeddings such as bert @xcite , which have been shown to achieve better performance across a wide range of natural language processing tasks than language models with static word embeddings. while it is nontrivial to use bert's bi-directional architecture and its masking mechanism for language modelling tasks, @xcite have recently shown how to obtain prediction values for bert and other architectures trained with masking loss. subword representations as used in bert may also help to compensate ocr-based errors, when only a few letters have been falsely recognized. on the downside, it is questionable whether the present corpus sizes of 300/500k token are large enough to obtain reliable estimates for the large number of bert's parameters. a potential solution is to rely on a bert model pre-trained by a large corpus, and to use the individual corpora to fine-tune the language model. though such fine tuning may enhance the predictions over the pre-trained model only, such an approach would mix norm-based and individual corpus information. the aim of the present study, in contrast, was to focus on the comparison of norm-based vs. strictly individual corpora, so we leave this extension for future work foot_2 .