T oronto CL at CMCL 2021 Shared Task: R o BERT a with Multi-Stage Fine-Tuning for Eye-Tracking Prediction (2021)

Abstract: Eye movement data during reading is a useful source of information for understanding language comprehension processes. In this paper, we describe our submission to the CMCL 2021 shared task on predicting human reading patterns. Our model uses RoBERTa with a regression layer to predict 5 eye-tracking features. We train the model in two stages: we first fine-tune on the Provo corpus (another eye-tracking dataset), then fine-tune on the task data. We compare different Transformer models and apply ensembling methods to improve the performance. Our final submission achieves a MAE score of 3.929, ranking 3rd place out of 13 teams that participated in this shared task.

## introduction eye-tracking data provides precise records of where humans look during reading, with millisecond-level accuracy. this type of data has recently been leveraged for uses in natural language processing: it can improve performance on a variety of downstream tasks, such as part-of-speech tagging @xcite , dependency parsing @xcite , and for cognitively-inspired evaluation methods for word embeddings @xcite . meanwhile, transformer-based language models such as bert @xcite and roberta @xcite have been applied to achieve state-of-the-art performance on many natural language tasks. the cmcl 2021 shared task aims to add to our understanding of how language models can relate to eye movement features. in this paper, we present our submission to this shared task, which achieves third place on the leaderboard. we first explore some simple baselines using token-level features, and find that these are already somewhat competitive with the final model's performance. next, we describe our model architecture, which is based on roberta (figure 1). we find that model ensembling offers a substantial performance gain over a single model. finally, we augment the provided training data with the publicly available provo eye-tracking corpus and combine them using a two-stage fine-tuning procedure; this results in a moderate performance gain. our source code is available at https://github. com/spoclab-ca/cmcl-shared-task . ## task description the shared task format is described in @xcite , which we will briefly summarize here. the task data consists of sentences derived from the zuco 1.0 and zuco 2.0 datasets; 800 sentences (15.7 tokens) were provided as training data and 191 sentences (3.5k tokens) were held out for evaluation. the objective is to predict five eye-tracking features for each token: the features are averages across multiple participants, and each feature is scaled to be in the range @xcite . the evaluation metric is the mean absolute error (mae) between the predicted and ground truth values, with all features weighted equally. since each team is allowed only a small number of submissions, we define our own train and test split to compare our models' performance during development. we use the first 600 sentences as training data and the last 200 sentences for evaluation during development. except for the submission results ( ## conclusion and future work we propose a simple approach to predict eyetracking features using the roberta model customized with a per-token regression head. our initial model uses the standard fine-tuning procedure; experiments show that the performance is further improved by model ensembling and domain adaptation by two-stage fine-tuning on an intermediate eye-tracking task. our best model achieves third place on the leaderboard. in future work, several avenues may be explored to further improve performance. first, we did not combine our feature engineering baseline with the roberta model -engineered features (such as frequency statistics or neurolinguistic norms) would provide the model with information not contained in roberta. second, we only experimented with a small subset of features from the provo corpus for domain adaptation, whereas it is not actually necessary for the auxiliary fine-tuning task to match the target task. thus, it may be possible to achieve better performance by fine-tuning on a different set of provo features, or a different dataset entirely.