CMCL 2021 Shared Task on Eye-Tracking Prediction (2021)

Abstract: Eye-tracking data from reading represent an important resource for both linguistics and natural language processing. The ability to accurately model gaze features is crucial to advance our understanding of language processing. This paper describes the Shared Task on Eye-Tracking Data Prediction, jointly organized with the eleventh edition of the Work- shop on Cognitive Modeling and Computational Linguistics (CMCL 2021). The goal of the task is to predict 5 different token- level eye-tracking metrics of the Zurich Cognitive Language Processing Corpus (ZuCo). Eye-tracking data were recorded during natural reading of English sentences. In total, we received submissions from 13 registered teams, whose systems include boosting algorithms with handcrafted features, neural models leveraging transformer language models, or hybrid approaches. The winning system used a range of linguistic and psychometric features in a gradient boosting framework.

## introduction/overview the ability of accurately modeling eye-tracking features is crucial to advance the understanding of language processing. eye-tracking provides millisecond-accurate records on where humans look, shedding lights on where they pay attention during their reading and comprehension phase (see the example in figure 1 ). the benefits of utilizing eye movement data have been noticed in various domains, including natural language processing and computer vision. not only can it reveal the workings of the underlying cognitive processes of language understanding, but the performance of computational models can also be improved if their inductive bias is adjusted using human cognitive signals such as eye-tracking, fmri, or eeg the film often achieves a mesmerizing poetry. data @xcite @xcite . thanks to the recent introduction of a standardized dataset @xcite , it is now possible to compare the capabilities of machine learning approaches to model and analyze human patterns of reading. in this shared task, we present the challenge of predicting eye word-level tracking-based metrics recorded during english sentence processing. we encouraged submissions concerning both cognitive modeling and linguistically motivated approaches (e.g., language models). all data files are available on the competition website. foot_0 ## related work research on naturalistic reading has shown that fixation patterns are influenced by the predictability of words in their sentence context @xcite . in natural language processing and psycholinguistics, the most influential account of the phenomenon is surprisal theory @xcite , which claims that the processing difficulty of a word is proportional to its surprisal, i.e., the negative logarithm of the probabil-ity of the word given the context. surprisal theory was the reference framework for several studies on language models and eye-tracking data prediction @xcite @xcite . these studies use the data from the dundee corpus @xcite , which consists of sentences from british newspapers with eye-tracking measurements from 10 participants, as one of the earliest and most popular benchmarks. later work on the topic found that the perplexity of a language model is the primary factor determining the fit to human reading times @xcite , a result that was confirmed also by the recent investigations involving neural language models such as gru networks @xcite and transformers @xcite @xcite . using an alternative approach, bautista and naval (2020) obtained good results for the prediction of eye movements with autoencoders. in addition to the zuco corpus used for this shared task (see section 4), there are several other resources of eye-tracking data for english. the ghent eye-tracking corpus (geco; @xcite is composed of the entire agatha christie's novel the mysterious affair at styles, for a total of 54, 364 tokens, it contains eye-tracking data from 33 subjects, both english native speakers (14) and bilingual speakers of dutch and english (19), and comes with the dutch counterpart. the provo corpus (luke and christianson, 2017) contains 55 short english texts about various topics, with 2.5 sentences and 50 words on average, for a total of 2, 689 tokens, and eye-tracking measures collected from 85 subjects. annotated eye-tracking corpora are also available for other languages, including german @xcite , hindi @xcite , japanese @xcite and russian (laurinavichyute et al., 2019), among others. ## task description in this shared task, we present the challenge of predicting eye-tracking-based metrics recorded during english sentence processing. the task is formulated as a regression task to predict the following 5 eye-tracking features for each token in the context of a full sentence: the goal of the task is to train a model which predicts these five eye-tracking features for each token in a given sentence. ## data we use the eye-tracking data recorded during normal reading from the freely available zurich cognitive language processing corpus (zuco; @xcite . zuco is a combined eye-tracking and eeg brain activity dataset, which provides anonymized records in compliance with an ethical board approval and as such it does not the eye-tracking data was recorded with an eye-link 1000 system in a series of naturalistic reading experiments. full sentences were presented at the same position on the screen one at a time. the participants read each sentence at their own reading speed. the reading material included sentences from movie reviews from the stanford sentiment treebank @xcite and a wikipedia dataset @xcite . for a detailed description of the data acquisition, please refer to the original publications. an example sentence is presented in figure 1 . we use the normal reading paradigms from zuco, i.e, task 1 and task 2 from zuco 1.0, and all tasks from zuco 2.0. we extracted the eyetracking data from all 12 subjects from zuco 1.0 and all 18 subjects from zuco 2.0. the dataset contains 990 sentences. all sentences were shuffled randomly before splitting into training and test sets. the training data contains 800 sentences, and the test data 190 sentences. ## evaluation in this section, we describe the evaluation procedure used to assess the submitted predictions of the participating teams. any additional data source was allowed to train the models, as long as it is freely available to the research community. for example, additional eyetracking corpora, additional features such as brain activity signals, pre-trained language models, etc. ## results in this section, we describe the prediction performance achieved by the participating teams. the official results of this shared task are presented in the difficulty of predicting the individual eyetracking features is analogous in all submitted systems. ffd is the most accurately predicted feature. this seems to suggest that the models are more capable to capture early processing stages of lexical access compared to late-stage semantic integration, indexed by trt and nfix. generally, the error for the three features representing reading times in milliseconds (ffd, gpt, and trt), is much lower than for nfix and fix-prop. the latter are the features with the most variance. the mean baseline results also reveal the same patterns. the features with lower variance achieve lower maes. the fixprop feature, representing how likely a word is to be fixated, might be more challenging to predict since it is more dependent on subject-specific characteristics. nevertheless, when comparing the maes of each eyetracking feature to the mean baseline, the systems achieve the largest improvement on this feature. ## outlook & conclusion we presented the results of the first shared task on predicting token-level eye-tracking features recorded during natural sentences reading. we hope the cmcl shared task makes a lasting contribution to the field of linguistic cognitive modelling by providing researchers with a standard evaluation framework and a high quality dataset. despite the limited size of the test set, many previously reached conclusions can now be tested more thoroughly and future models can be compared on a shared benchmark. for future editions of this shared task, we see the following improvement opportunities: (1) providing an official development set during the training phase; (2) using additional metrics for assessment, such as r 2 to achieve a better understanding of the submitted models; (3) extending the dataset to include additional eye-tracking data from other english corpora, as well as including data from other languages such as dutch or russian (e.g., @xcite .