H k A msters at CMCL 2022 Shared Task: Predicting Eye-Tracking Data from a Gradient Boosting Framework with Linguistic Features (2022)

Abstract: Eye movement data are used in psycholinguistic studies to infer information regarding cognitive processes during reading. In this paper, we describe our proposed method for the Shared Task of Cognitive Modeling and Computational Linguistics (CMCL) 2022 - Subtask 1, which involves data from multiple datasets on 6 languages. We compared different regression models using features of the target word and its previous word, and target word surprisal as regression features. Our final system, using a gradient boosting regressor, achieved the lowest mean absolute error (MAE), resulting in the best system of the competition.

## introduction this year's cognitive modeling and computational linguistics (cmcl) workshop proposed a shared task focused on eye-tracking data prediction @xcite . differently from the last edition @xcite , the 2022 shared task includes two subtasks: "predict eye-tracking features for sentences of the 6 provided languages" and "predict eye-tracking features for sentences from a new surprise language". in this paper, we present our proposed method for the first subtask. in this task, the teams were asked to predict 4 eye-tracking features for 6 different languages (chinese, dutch, english, german, hindi, and russian); the features were: first fixation duration (ffd, which refers to the duration of the first fixation on the prevailing word), the standard deviation of the ffd across readers, total reading time (trt, which refers to the sum of all fixation durations on the current word, including regressions), and the standard deviation of trt across readers. one of the challenging aspects of this task is the substantially different nature of these languages; they belong to different language families or different branches within the same family (i.e., germanic, balto-slavic, indo-iranian, sino-tibetan) and 4 different writing systems are involved (i.e., latin alphabet, cyrillic alphabet, devanagari abugida, and logograms). therefore, we proposed a unified method that could be applied to account for the similarities and differences exhibited in the datasets of these 6 languages; this method includes regression features of the target word and of its previous word, and the surprisal for the target word within the context. our codes are shared on github at: https: //github.com/laviniasalicchi/ hkamsters_cmcl2022. ## related work eye movement data provide valuable evidence regarding the cognitive processes underlying reading, and thus revealing how language is elaborated in our brain in every aspect, from morphology @xcite to syntax @xcite to semantics @xcite . since the early studies published in the last century, several studies have revealed that some features of the words themselves may influence language processing and, consequently, reading behavior; these features include word position, word length, word frequency, and the number of syllables within the word @xcite . in addition, the spillover effect @xcite infers that the cognitive load of a word due to its frequency and length @xcite may influence the processing of its following word. considering the multilingual nature of this task, in addition to the aforementioned features, we also included whether the word is all in uppercase, and whether it begins with a capital letter. one additional factor that influences language comprehension is the sentence-level predictability of a word given the previous context @xcite , and in recent years, with the growth of computational linguistics, some attempts to model this kind of dynamic have been successfully achieved using the surprisal (i.e., the negative logarithm of the probability of encountering a word given the context) computed by language models @xcite @xcite . in the last year's shared task, a regression model was proposed using the following features: twoword features (i.e., word length and word frequency), the cosine similarity between the vector representing the target word and the vector representing the sentence context, and the surprisal computed word-by-word @xcite . however, @xcite showed that given the overlaps in the information conveyed by cosine similarity and the surprisal, the latter alone is sufficient for the effective modeling of eye movements. furthermore, in frank's model the frequency and length of the word preceding the target one are included in the regression for modeling the spillover effect. for these reasons, we modified the previously proposed method, increasing the number of wordspecific features, and excluding the cosine similarity in our system. ## datasets the shared task is formulated as a regression task to predict 2 eye-tracking features and their corresponding standard deviation across readers: (1) ffd; (2) the standard deviation of ffd across readers; (3) trt; and (4) the standard deviation of trt across readers. subtask 1 (multilingual prediction) requires systems to predict these four eye-tracking features of words in 6 provided languages. the dataset includes materials from 8 openly available eye movement corpora: data statistics are given in ## methodology in this section, we introduce the selected features, inspired by psycholinguistic studies relying on eyetracking data, and the investigated regression algorithms. the same set of features was used for each regression model. ## results and discussion to use a single model that could be applied to multilingual data, we selected the model with generally better performance. lgb, which is the gradient boosting regressor with the regression feature interacting, provided the best predictions. lgb not only had the lowest mae, but also achieved the best results in terms of mse, r2, pearson correlation, and spearman correlation. lgb is the best performed model for hindi. brr and rr are the second and third best models, but the performance gap is rather marginal. considering how regressors accounted for each language dataset, we present the lowest mae values for each feature and each language in given the differences in the amount of data among language datasets, our comparison mainly follows the coefficient of variation, which reveals that for ffdavg, english, german, and hindi were the languages for which our system performed better, followed by dutch, russian and chinese. for trtstd, the better performances were on the datasets of hindi, chinese, and russian, whereas the most difficult portions of the dataset for this feature were those in english, dutch, and german. finally, regarding the errors for ffdstd, english finally, we performed an ablation study for the german dataset, in order to examine the contribu despite some minor performance drop (especially for trt), using capitalization generally improves the evaluation metrics. ## conclusions in this paper, we described the system we proposed for the cmcl2022 shared task -subtask 1 on multilingual data. using a gradient boosting regressor with features of the target words as well as their previous word, and the surprisal between the target word and the previous context as regression features, we predicted two eye-tracking features and two standard deviations: first fixation duration, total reading time, and their standard deviations across readers. despite the multilingual nature of this task, we were able to reach our goal of creating a unified system capable of modeling the human reading behavior in 6 substantially different languages. our results showed a tendency of better performances with ffd related features than with trt related ones. this may partly reflect the fact that in our system, more word-level hand-crafted features were included, which could favor this token-level prediction task, given that ffd is often assumed to reflect lexical information processing, whereas trt may be related to a later stage of language processing related to information-structural integration.