PIHK ers at CMCL 2021 Shared Task: Cosine Similarity and Surprisal to Predict Human Reading Patterns. (2021)

Abstract: Eye-tracking psycholinguistic studies have revealed that context-word semantic coherence and predictability influence language processing. In this paper we show our approach to predict eye-tracking features from the ZuCo dataset for the shared task of the Cognitive Modeling and Computational Linguistics (CMCL2021) workshop. Using both cosine similarity and surprisal within a regression model, we significantly improved the baseline Mean Absolute Error computed among five eye-tracking features.

## introduction the shared task proposed by the organizers of the cognitive modeling and computational linguistics workshop @xcite requires participant to create systems capable of predicting eye-tracking data from the zuco dataset @xcite . creating systems to efficiently predict biometrical data may be useful to make prediction about linguistic materials for which we have few or none experimental data, and to make new hypothesis about the internal dynamics of cognitive processes. the approach we propose relies mainly on two factors that have been proved to influence language comprehension: i.) the semantic coherence of a word with the previous ones @xcite and ii.) its predictability from previous context @xcite . we model the first factor with the cosine similarity @xcite between the distributional vectors, representing the context and the target word, produced by different distributional semantic models (dsm) @xcite . we compared 10 state-of-the-art word embedding models, and two different approaches to compute the context vector. we model the predictability of a word within the context with the word-by-word surprisal computed with 3 of the above mentioned models @xcite . finally, cosine similarity and surprisal are combined in different regression models to predict eye tracking data. ## related works different word embedding models (glove, word2vec, wordnet2vec, fasttext, elmo, bert) have been evaluated in the framework proposed by @xcite . the evaluation is based on the model capability to reflect semantic representations in the human mind, using cognitive data in different datasets for eye-tracking, eeg, and fmri. word embedding models are used to train neural networks on a regression task. the results of their analyses show that bert, elmo, and fasttext have the best prediction performances. regression models with different combinations of cosine similarity and surprisal, to predict (and further study the cognitive dynamics beneath) eye movements have been created by @xcite , who claims that, since word embeddings are based on co-occurrences, semantic distance may actually represent word predictability, rather than semantic relatedness, and that previous findings showing correlations between reading times and semantic distance were actually due to a confound between these two concepts. in his work, he uses linear regression models testing different surprisal measures, and excluding it. the results show that when surprisal is factored out, the effects of semantic similarity on reading times disappear, proving thus the existence of an interplay between the two elements. ## results and discussion spearman's correlations between eye tracking features and cosine similarity showed that best performances are reached by vectors produced by bert layer 22 cls context (mean correlation over eye tracking features on the three datasets: -0.62), while best correlations between eye tracking data and surprisal are reached by gpt2-xl (mean correlation over eye tracking features on the three datasets: 0.40). these results led us to select as features for regression model: cosine similarity between vectors computed by bert 22 cls and surprisal computed by gpt2-xl. we also tested the cosine similarity between vectors computed by gpt2-xl, to have a comparison with a regression model with features produced by the same model. while performing regression model selection comparing 9 models from scikit-learn, we also tried different combinations of features. therefore, we tested again the 9 regression models with scikit-learn, this time using nfix, ffd, trt, gpt, word lenght and word frequency as features, in every possible permutation (one per time, pairs of features, etc.). a lower mae on fixprop on training data has been obtained using a random forest method with nfix, trt, and gpt, reaching a mae of 3.15. the improvements of the final model over the baseline suggest that the information conveyed by the cosine similarity and the surprisal contributes in modeling the cognitive processing beneath reading. our results are consistent with @xcite and @xcite findings about the relation between cosine similarity and eye movements data, as well as with @xcite and @xcite , who found surprisal to be useful in predicting reading times. anyway, our model performance shows that taking into account both the computational measures benefits the modeling. even if @xcite rises an interesting issue about the interplay between the information included in word embeddings and the one provided by the suprisal computed by language models, our results keep us from fully agree with his observations: since the joined model performed better that the ones taking into account only cosine similarity or only surprisal, it is obvious that the two measures convey exclusive and useful information, even if it is more than plausible that they share some kind of information to some extent. in summary, we used a two-step approach: i.) the final model to predict nfix, ffd, gpt, and trt in test data was a gradient boosting regressor having as features the cosine similarity between the cls vector (bert) and the target word embedding, gpt2-xl surprisal, word length and word frequency; ii.) the predicted values of nfix, gpt, and trt were used in a random forest to predict fixprop. the shared task final results over the test data, revealed that our model had an average mae of 4.3877 over all eye tracking features (the baseline was 7.3699, while the best model reached a mae of 3.8134). ## conclusions in this paper we described the system we proposed in the cmcl2021 "shared task: predicting human reading patterns". we were required to create a model capable of predicting number of fixations, first fixation duration, total reading time, go-past time, and fixation proportion of each word in the zuco dataset. we proposed a regression model using word length and word frequency, combined with two elements that are proved to influence reading processing: the semantic coherence and the predictability of a word within the context. to compute these last two regression features we used the cosine similarity between the vector representing the context and the word embedding of the target word, and the surprisal computed by language models, respectively. we selected the models to produce the vectors and to compute the surprisal calculating the spearman correlation between the cosine similarity and the eye tracking data, and between the surprisal and the same data. we then used the best cosine similarity and surprisal within a regression model, selected among 9 possible models. our results outperformed the baseline, with a average mae among eye tracking features just 0.5743 higher than the best model in the competition. our model may be improved exploring new types of regressors and word embeddings, and including new textual features such as sentence length and information regarding words immediately preceding the target ones.