CMCL 2022 Shared Task on Multilingual and Crosslingual Prediction of Human Reading Behavior (2022)

Abstract: We present the second shared task on eye-tracking data prediction of the Cognitive Modeling and Computational Linguistics Workshop (CMCL). Differently from the previous edition, participating teams are asked to predict eye-tracking features from multiple languages, including a surprise language for which there were no available training data. Moreover, the task also included the prediction of standard deviations of feature values in order to account for individual differences between readers.A total of six teams registered to the task. For the first subtask on multilingual prediction, the winning team proposed a regression model based on lexical features, while for the second subtask on cross-lingual prediction, the winning team used a hybrid model based on a multilingual transformer embeddings as well as statistical features.

## introduction the benefits of eye movement data for machine learning have been assessed in various domains, including nlp @xcite @xcite and computer vision @xcite @xcite . eyetracking provides millisecond-accurate records on where humans look when they are reading and are useful in explanatory research of language processing. eye movements depend on the stimulus and are therefore language-specific, but there are also universal tendencies that have been observed across languages @xcite . modelling human reading has been researched extensively in psycholinguistics @xcite @xcite . in nlp, eye-tracking prediction has been used to determine linguistic complexity @xcite or to analyze language models' ability to account for measures of human reading effort @xcite . being able to accurately predict eye-tracking features across languages will advance this field and will facilitate comparisons between models and the analysis of their varying capabilities. in this shared task, we address the challenge of predicting eye-tracking features recorded during sentence processing of multiple languages. we are interested in both cognitive modelling approaches as well as linguistically motivated approaches (i.e., language models). this shared task is hosted on codalab, where the instructions and pre-processed eye-tracking datasets are available. 1 compared to the cmcl 2021 shared task on eye-tracking prediction @xcite , we introduce two major changes: ## task description the shared task is formulated as a regression task to predict 2 eye-tracking features and the corresponding standard deviation across readers for each word: ## evaluation in this section, we describe the evaluation procedure used to assess the submitted predictions of the participating teams. any additional data source was allowed to train the models, as long as it is freely available to the research community. for example, additional eyetracking corpora, additional features such as brain activity signals, pre-trained language models, etc. ## participating teams & systems six teams and a total of 37 participants registered on the competition website. all six teams submitted their predictions during the evaluation phase for subtask 1. four of the teams also submitted predictions for subtask 2. each team was allowed three submissions for each subtask during the evaluation phases. finally, 5 teams published system description papers outlining their approaches (see ## results in this section, we describe the prediction performance achieved by the participating teams. the official results of this shared task are presented in tables 3 and 4 for subtask 1 and 2, respectively. the best results for the first subtask on multilingual prediction were achieved by team hkamsters with language-specific regression models based on word-level features such as word length, word frequency, and surprisal scores estimated with gpt-2 @xcite . for the second subtask on cross-lingual prediction, the winning team (poirot) used a zero-shot hybrid model based on a multilingual transformer embeddings as well as statistical features. ## outlook & conclusion we presented the results of the second shared task on predicting token-level eye-tracking features recorded during natural reading of sentences or longer text spans. in this second edition, we focused on multilingual and crosslingual prediction. we hope the cmcl shared task makes a lasting contribution to the field of linguistic cognitive modelling by providing researchers with a standard evaluation framework and a high quality dataset. despite the limited size of the training and test sets as well as the diversity of text domains within the eye-tracking corpora, many previously reached conclusions can now be tested more thoroughly and future models can be compared on a shared multilingual benchmark.