Looking for a Role for Word Embeddings in Eye-Tracking Features Prediction: Does Semantic Similarity Help? (2021)

Abstract: Eye-tracking psycholinguistic studies have suggested that context-word semantic coherence and predictability influence language processing during the reading activity. In this study, we investigate the correlation between the cosine similarities computed with word embedding models (both static and contextualized) and eye-tracking data from two naturalistic reading corpora. We also studied the correlations of surprisal scores computed with three state-of-the-art language models. Our results show strong correlation for the scores computed with BERT and GloVe, suggesting that similarity can play an important role in modeling reading times.

## introduction eye-tracking data recorded during reading provide invaluable evidence about the factors influencing language comprehension. research in computational modeling has particularly focused on two factors: i.) the semantic coherence of a word with the rest of the sentence @xcite @xcite , measured via semantic similarity metrics and ii.) its predictability from previous context, as measured by surprisal @xcite . intuitively, words that have low semantic coherence and low in-context predictability (i.e., high surprisal) induce longer reading times. in distributional semantics @xcite , words and their sentence contexts are represented with dense vectors called embeddings and produced by distributional semantic models (dsm). in this paper, we modeled semantic coherence with the cosine similarity between the embeddings of words and their sentence contexts, and then we tested the correlation of the metric with the eye-tracking measures annotated on the geco and provo corpora. we analyzed the correlations for the similarity computed with 10 different embedding models (both static and contextualized), as well as for surprisal scores computed with several state-of-the-art neural language models. among all the features under investigation, the similarity scores obtained with bert and glove obtained the best correlations across features in both the benchmark corpora. a series of contributions explored the role of surprisal in modeling reading times in naturalistic settings, coming to the general conclusion that the predictive power is strongly related to the language model quality, i.e. models with better perplexity perform better (smith and @xcite . later work explored the most recent neural models, including lstm (van schijndel and linzen, 2018), gru @xcite , transformers @xcite and gpt-2 @xcite , basically confirming this relationship. 1 early studies had also found correlations between semantic distance, computed by word embeddings, and eye-tracking features in reading processes @xcite . however, the more recent work by @xcite pointed out that, since word embeddings are based on co-occurrences, semantic distance may actually represent word predictability, rather than semantic relatedness, and that those early findings were actually due to a confound between these two concepts. to test this hypothesis, the author used linear regression models with and without surprisal, testing 5 surprisal measures. the results show that the effects of similarity on reading times disappear when surprisal is factored out, thereby proving the existence of a complex interplay between the two factors. frank's experiments were carried out in a naturalistic reading setting and, to our knowledge, there have been no eye-tracking studies with controlled stimuli investigating a possible separate effect of the two components (for example, by comparing the fixation patterns of words that have low predictability, but different degrees of coherence with the sentence or with the discourse context). ## results and discussion looking at the correlations results, it is clear that every model performed better on provo. one possible explanation for this difference is that geco eye-tracking data are recorded on participants reading a literary text, while provo materials are online news articles, science magazines and only partially short text from works of fiction. the consequence is a difference in the syntactic complexity of sentence structure and in the frequency of words. this gap implies that the modeling of geco contexts is less directly reducible to an additive fashion of processing, and, most importantly, is more likely to find out of vocabulary words in geco, rather than in provo. another aspect that is quite evident are the similar correlation values among different eye-tracking features. this aspect is not surprising: in the original datasets of geco and provo, it can be noticed that many words show the same value for the total reading time and the first fixation duration. this happens when i) the word is not read (0 ms for both the features); ii) the word is read only once (total reading time and first fixation duration overlap). also regarding the similar values of the correlations between similarity and number of fixations and between similarity and total reading times, taking into account the original data gives us an explanation of the results: since the total reading time is computed summing the duration of all the multiple fixations, the higher the number of fixation, the higher the total reading time, leading to a similar tendency in the values of the two features. for these reasons, the total reading time may be considered as a "bridge" field, that holds close relations with both first fixation duration and number of fixations, justifying the similar correlation values in our results. ## conclusions and ongoing work in this paper, we have used contextualized and non-contextualized dsms to compute the cosine between a target word and the previous sentence context. our results show that cosine similarity is able to achieve very high correlations with the eyetracking metrics of geco and provo, especially with the bert and glove models, providing further evidence that semantic coherence is potentially very useful in modeling reading times. furthermore, we computed word-by-word surprisal using bert, gpt2-xl, and neural complexity. among the language models, the best results have been achieved by gpt2-xl, confirming the previous findings that transformers are very good at modeling sentence processing metrics @xcite @xcite . however, the absolute value of correlation is lower than the one obtained with cosine similarity scores: for example, the mean correlation achieved on provo with the cosine similarity between vectors produced by bert is -0.71, while the correlation between eye tracking features and the surprisal computed by the same model is 0.24. the comparison between correlations reached by cosine similarity and surprisal may lead us to the conclusion that semantic coherence is a stronger predictor of eye-tracking features than word predictability. however, given the significant degree of correlation between cosine similarity and surprisal, further investigations are needed to disentangle the two factors. our next step will be to include transformersbased surprisal and vector-based cosine similarity in a large-scale regression study to predict eye tracking features, in order to ensure a close comparison with the experimental setting of @xcite , and to investigate if semantic similarity models can actually play a distinct role from surprisal in the prediction of reading times. differently from @xcite , we plan to test with several regression models, from a simple linear regression to more advanced regression models (e.g. gradient boosting, multilayer perceptron etc.), and with different word embedding models, in order to account for the different types of semantic similarity computed by static and contextualized embeddings.