LAST at CMCL 2021 Shared Task: Predicting Gaze Data During Reading with a Gradient Boosting Decision Tree Approach (2021)

Abstract: A LightGBM model fed with target word lexical characteristics and features obtained from word frequency lists, psychometric data and bigram association measures has been optimized for the 2021 CMCL Shared Task on Eye-Tracking Data Prediction. It obtained the best performance of all teams on two of the five eye-tracking measures to predict, allowing it to rank first on the official challenge criterion and to outperform all deep-learning based systems participating in the challenge.

## introduction this paper describes the system proposed by the laboratoire d'analyse statistique des textes (last) for the cognitive modeling and computational linguistics (cmcl) shared task on eye-tracking data prediction. this task is receiving more and more attention due to its importance in modeling human language understanding and improving nlp technology @xcite . as one of the objectives of the organizers is to "compare the capabilities of machine learning approaches to model and analyze human patterns of reading" ( https://cmclorg.github.io/ shared_task ), i have chosen to adopt a generic point of view with the main objective of determining what level of performance can achieve a system derived from the one i developed to predict the lexical complexity of words and polylexical expressions @xcite . that system was made up of a gradient boosting decision tree prediction model fed with features obtained from word frequency lists, psychometric data, lexical norms and bigram association measures. if there is no doubt that predicting lexical complexity is a different problem, one can think that the features useful for it also play a role in predicting eye movement during reading. the next section summarizes the main characteristics of the challenge. then the developed system is described in detail. finally, the results in the challenge are reported along with an analysis performed to get a better idea of the factors that affect the system performance. ## data and task the eye-tracking data for this shared task were extracted from the zurich cognitive language processing corpus (zuco 1.0 and zuco 2.0, @xcite . it contains gaze data for 991 sentences read by 18 participants during a normal reading session. the learning set consisted in 800 sentences and the test set in 191 sentences. the task was to predict five eye-tracking features, averaged across all participants and scaled in the range between 0 and 100, for each word of a series of sentences: (1) the total number of fixations (nfix), (2) the duration of the first fixation (ffd), (3) the sum of all fixation durations, including regressions (trt), (4) the sum of the duration of all fixations prior to progressing to the right, including regressions to previous words (gpt), and (5) the proportion of participants that fixated the word (fixprop). these dependent variables (dvs) are described in detail in @xcite . the submissions were evaluated using the mean absolute error (mae) metric and the systems were ranked according to the average mae across all five dvs, the lowest being the best. as the dvs are of different natures (number, proportion and duration), their mean and variance are very different. the mean of fixprop is 21 times greater than that of ffd and its variance 335 times. furthermore, while nfix and fixprop were scaled independently, ffd, gpt and trt were scaled together. for that reason, the mean and dispersion of these three measures are quite different: ffd must necessarily be less than or equal to trt and gpt foot_0 . these two factors strongly affect the importance of the different dvs in the final ranking. ## conclusion the system proposed for the 2021 cmcl shared task on eye-tracking data prediction was particularly effective, obtaining the first place in the challenge, but it must be kept in mind that the system that came second is superior to it for three of the five dvs. the analyzes carried out to understand its pros and cons indicate that optimizing the light-gbm parameters is quite beneficial to it as well as the different sets of features derived from corpora and behavioral data, including bigram ams which, to my knowledge, have never been employed for this type of task. it would have been interesting to relate these observations to the psycholinguistic literature on the factors that influence eye fixations, but this is unfortunately not possible here, for lack of space. in addition, this would first require deepening the ablation analyzes by simultaneously considering several feature sets. for instance, the lack of usefulness of the pos-tags could simply result from the links (at least partial) between them and the frequency and length of the tokens. likewise, some of the bigram ams are relatively sensitive to the frequency of the words that compose them (e.g., the t-score favors frequent bigrams which are usually composed of frequent words). it is thus highly probable that some of the features in the different sets (frequencies, behavioral data...) are redundant and can be removed without impairing the performance of the system. this is a potential development path.