Team O hio S tate at CMCL 2021 Shared Task: Fine-Tuned R o BERT a for Eye-Tracking Data Prediction (2021)

Abstract: This paper describes Team Ohio State’s approach to the CMCL 2021 Shared Task, the goal of which is to predict five eye-tracking features from naturalistic self-paced reading corpora. For this task, we fine-tune a pre-trained neural language model (RoBERTa; Liu et al., 2019) to predict each feature based on the contextualized representations. Moreover, motivated by previous eye-tracking studies, we include word length in characters and proportion of sentence processed as two additional input features. Our best model strongly outperforms the baseline and is also competitive with other systems submitted to the shared task. An ablation study shows that the word length feature contributes to making more accurate predictions, indicating the usefulness of features that are specific to the eye-tracking paradigm.

## introduction behavioral responses such as eye-tracking data provide valuable insight into the latent mechanism behind real-time language processing. based on the well-established observation that behavioral responses reflect processing difficulty, cognitive modeling research has sought to accurately predict these responses using theoretically motivated variables (e.g. surprisal; @xcite . earlier work in this line of research has introduced incremental parsers for deriving psycholinguisticallymotivated variables (e.g. @xcite , while more recent work has focused on evaluating the capability of neural language models to predict behavioral responses @xcite . the cmcl 2021 shared task on eye-tracking data prediction @xcite provides an appropriate setting to compare the predictive power of different approaches using a standardized dataset. according to the task definition, the goal of the shared task is to predict five eye-tracking features from naturalistic self-paced reading corpora, namely the zurich cognitive language processing corpus 1.0 and 2.0 (zuco 1.0 and 2.0; @xcite . these corpora contain eyetracking data from native speakers of english that read select sentences from the stanford sentiment treebank @xcite and the wikipedia relation extraction corpus @xcite . the five eye-tracking features to be predicted for each word, which have been normalized to a range between 0 and 100 and then averaged over participants, are as follows: in this paper, we present team ohio state's approach to the task of eye-tracking data prediction. as the main input feature available from the dataset is the words in each sentence, we adopt a transfer learning approach by fine-tuning a pre-trained neural language model to this task. furthermore, we introduce two additional input features motivated by previous eye-tracking studies, which measure word length in characters and the proportion of sentence processed. our best-performing model outperforms the mean baseline by a large margin in terms of mean absolute error (mae) and is also competitive with other systems submitted to the shared task. 2 model description our model relies primarily on the transformerbased pre-trained language model roberta @xcite for contextualized representations of each word in the input sentence. 1 however, since roberta uses byte-pair encoding @xcite to tokenize each sentence, there is a mismatch between the number of output representations from roberta and the number of words in each sentence. in order to address this issue, the model uses the representation for the first token associated with each word to make predictions. for example, if byte-pair encoding tokenizes the word carlucci into car, lu, and cci, the representation for car is used to make predictions for the entire word carlucci. 2 additionally, two input features based on previous eye-tracking studies are included in the model. the first is word length measured in characters (wlen), which captures the tendency of readers to fixate longer on orthographically longer words. the second feature is proportion of sentence processed (prop), which is calculated by dividing the current index of the word by the number of total words in each sentence. this feature is intended to take into account any "edge effects" that may 1 although other word representations could be used within our model architecture, the use of roberta was motivated by its state-of-the-art performance on many nlp tasks. the robertabase and robertalarge variants were explored in this work, which resulted in two different models. we used the implementation made available by huggingface ( https: //github.com/huggingface/transformers ). ## results and discussion to evaluate the contribution of the wlen and prop features, an ablation study was conducted using the roberta base model. in addition to showing how useful wlen and prop information is for predicting eye-tracking features, the analysis was also thought to reveal whether or not such information is already contained within the roberta representations. the two input features were ablated by simply replacing them with zeros during inference, which allowed a clean manipulation of their contribution to the final predictions. the results in the ablation of the wlen feature resulted in a more notable difference in four out of five eyetracking features. this indicates that information about orthographic length is both useful for eyetracking data prediction and also orthogonal to the information captured by the roberta representations. this may partially be explained by roberta's use of byte-pair encoding, which can result in many short tokens for a given word (e.g. tokens car, lu, cci for the word carlucci). since only the first token was used by the current models to represent each word, explicitly including information about word length seems to have contributed to making more accurate predictions. more generally, this highlights the utility of incorporating features that are specific to eye-tracking, which may not be inherent in high-capacity language models trained for a different objective. ## conclusion in this paper, we present our approach to the cmcl 2021 shared task on eye-tracking data prediction. our models primarily adopt a transfer learning approach by employing a feedforward neural network to predict eye-tracking features based on contextualized representations from a pre-trained language model. additionally, we include two input features that have been known to influence eye movement, which are word length in characters (wlen) and proportion of sentence processed (prop). our best model based on roberta base strongly outperforms the mean baseline and is also competitive with other systems submitted to the shared task. a follow-up ablation study shows that the wlen feature contributed to making more accurate predictions, which indicates that explicitly incorporating features specific to the eye-tracking paradigm can complement high-capacity language models on this task.