K on T ra at CMCL 2021 Shared Task: Predicting Eye Movements by Combining BERT with Surface, Linguistic and Behavioral Information (2021)

Abstract: This paper describes the submission of the team KonTra to the CMCL 2021 Shared Task on eye-tracking prediction. Our system combines the embeddings extracted from a fine-tuned BERT model with surface, linguistic and behavioral features, resulting in an average mean absolute error of 4.22 across all 5 eye-tracking measures. We show that word length and features representing the expectedness of a word are consistently the strongest predictors across all 5 eye-tracking measures.

## introduction the corpora zuco 1.0 and zuco 2.0 by @xcite contain eye-tracking data collected in a series of reading tasks on english materials. for each word of the sentences, five eyetracking measures are recorded: 1) the number of fixations (nfix), 2) the first fixation duration (ffd), 3) the go-past time (gpt), 4) the total reading time (trt), and 5) the fixation proportion (fixprop). providing a subset of the two corpora, the cmcl 2021 shared task @xcite requires the prediction of these eye-tracking measures based on any relevant feature. to tackle the task, we conduct a series of experiments using various combinations of bert embeddings @xcite and a rich set of surface, linguistic and behavioral features (slb features). our experimental setting enables a comparison of the potential of bert and the slb features, and allows for the explainability of the system. the best performance is achieved by the models combining word embeddings extracted from a fine-tuned bert model with a subset of the slb features that are the most predictive for each eye-tracking measure. overall, our model was ranked 8th out of 13 models submitted to the shared task. our main contributions are the following: 1) we show that training solely on slb features provides better results than training solely on word embeddings (both pre-trained and fine-tuned ones). ## describing eye-tracking measures to explore the impact of linguistic and cognitive information on eye-movements in reading tasks, we extract a set of surface, linguistic, behavioral and bert features, as listed in surface features given the common finding that surface characteristics, particularly the length of a word, influence fixation duration @xcite , we compute various surface features at word and sentence level (e.g., word and sentence length). ## predicting eye-tracking measures we conduct three experiments using different feature combinations, and experiment with three model architectures. the models' parameters are experimentally defined. first, we train a linear regression model (lr). second, we train a decision tree regressor (dt) with the mse (mean squared error) criterion and a maximum depth of 7. last, we train a random forest regressor (rf) with the mse criterion, 15 estimators and a maximum depth of 7. before training the models, all categorical feature values are one-hot-encoded and all numeric values are normalized within the range [0, 1]. ## conclusion we presented a system of eye-tracking feature prediction which combines bert with a rich set of surface, linguistic and behavioral (slb) features. overall, our three studies indicate that including not only semantic properties that can be directly extracted from text, such as embeddings and surprisal score, but also measures reflecting behavioral (e.g., dominance and age of acquisition) and surface properties (word and sentence length) has a positive impact on the performance of our models in predicting eye-tracking data.