Team ÚFAL at CMCL 2022 Shared Task: Figuring out the correct recipe for predicting Eye-Tracking features using Pretrained Language Models (2022)

Abstract: Eye-Tracking data is a very useful source of information to study cognition and especially language comprehension in humans. In this paper, we describe our systems for the CMCL 2022 shared task on predicting eye-tracking information. We describe our experiments withpretrained models like BERT and XLM and the different ways in which we used those representations to predict four eye-tracking features. Along with analysing the effect of using two different kinds of pretrained multilingual language models and different ways of pooling the token-level representations, we also explore how contextual information affects the performance of the systems. Finally, we also explore if factors like augmenting linguistic information affect the predictions. Our submissions achieved an average MAE of 5.72 and ranked 5th in the shared task. The average MAE showed further reduction to 5.25 in post task evaluation.

## introduction and motivation in the last decade that has seen rapid developments in ai research, the emergence of the transformer architecture @xcite marked a pivotal point in natural language processing (nlp). fine-tuning pretrained language models to work on various downstream tasks has become a dominant method of obtaining state-of-the-art performance in different areas. their capability to capture linguistic knowledge and learn powerful contextual word embeddings @xcite have made the transformer based models the work-horses in many nlp tasks. pretrained models like the multilingual bert @xcite and xlm @xcite have also shown state-of-the-art performance on cross-lingual understanding tasks @xcite . in some cases like machine translation, there are even claims that deep learning systems reach translation qualities that are comparable to professional translators @xcite . language processing and its links with cognition is a very old research problem which has revealed how cognitive data (eg. gaze, fmri) can be used to investigate human cognition. attempts at using computational methods for such studies @xcite have also shown encouraging results. however recently, there have been a number of works that have tried to incorporate human cognitive data collected during reading for improving the performance of nlp systems @xcite . the cmcl 2022 shared task of multilingual and cross-lingual prediction of human reading behavior @xcite explores how eye-gaze attributes can be algorithmically predicted given reading data in multilingual settings. informed by the previous attempts at using pretrained multilingual language models to predict human reading behavior @xcite we experiment with multilingual bert and xlm based models to test which fares better in this task. for the experiments with the pretrained models, we use the trained weights from huggingface @xcite and perform the rest of our experiments using pytorch foot_0 . inspired by the psycholinguistic research on investigating context length during processing (wochna and juhasz, 2013), we experiment how different contexts affect model performance. finally, we merged the principles of the "classical" approach of feature-based prediction with the pretrained-language model based prediction for further analysis. in the following sections, we present our results from a total of 48 different models. ## task description the cmcl 2022 shared task of multilingual and cross-lingual prediction of human reading behavior frames the task of predicting eye-gaze attributes associated with reading sentences as a regression task. the data for the task was comprised of eye movements corresponding to reading sentences in six languages (chinese, dutch, english, german, hindi, russian). the training data for the task contained 1703 sentences while the development set and test set contained 104 and 324 sentences respectively. the data was presented in a way such that for each word in a sentence there were four associated eye-tracking features in the form of the mean and standard deviation scores of the total reading time (trt) and first fixation duration (ffd). the features in the data were scaled in the range between 0 and 100 to facilitate evaluation via the mean absolute average (mae). ## experiments a total of 48 models of different configurations were trained with the data provided for the shared task. the different configurations used to construct the models are based on intuition and literature survey. thee models were primarily categorized as system-1 (sys1) and system-2 (sys2) models. for some word corresponding to a sentence in the dataset, system-1 models provided no additional context information. system-2 models on the other hand, contained the information of all the words in the sentence that preceded the current word, providing additional context. this setting was inspired by works @xcite on how context is used by language models. all systems under the system-1/2 labels were further trained as a bert (bert) based system or a xlm (xlm) based system. bert embeddings were previously used by @xcite for the eye-tracking feature prediction task in cmcl 2021. corresponding to each such language models (bert and xlm), the impact of different fine-tuning strategies @xcite on system performance was studied. hence, for one setting, only the contextualized word representation (cwr) was utilized by freezing the model weights and putting a learnable regression layer on top of the model output layer (classifier). alternatively, the models were fine-tuned with the regression layer on top of them (whole). this setting is similar to the one used by @xcite . however in our case, we experiment with a bert and xlm pretrained model. additionally, we also performed experiments with pooling strategies for the layer representations by either using the final hidden representation of the first sub-word encoding of the input (first) or aggregating the representations of all sub-words using mean-pooling (mean) or sum-pooling (sum). the rationale behind using different pooling strategies was to have a sentence-level representation of the input tokens. the impact of different pooling strategies has previously been studied @xcite for different problems. in this paper, we analyze the effect of pooling featurespace embeddings in the context of eye-tracking feature prediction. finally, for the experiments where we augmented additional lexical features (augmented) to the neural features for regression, we used word length and word-frequency as the additional information following @xcite . constructing the experiments in this manner provided us with models with a diverse set of properties and in turn provided insights into how well the model behaves when all other things stay the same, and only one aspect of learning is changed. ## results the results corresponding to the top 10 systems based on the experiments described above are shown in an analysis of the models based on the different experimental configurations are described in the ffd_avg ffd_std trt_avg trt_std 0.194 0.403 0.637 0.489 however, it should be noted that 12 out of the first 20 best performing models were system-2 models. hence we posit that although the availability of the full sentence context is a factor for having more efficient systems, independently the factor does not seem to boost the overall performance much. ## conclusion in this paper, we have described our experiments with different kinds of models that were trained on the data provided for this shared-task. we have identified five ways in which we can make better systems to predict eye-tracking features based on eye-tracking data from a multilingual corpus. first, the experiments demonstrate that the inclusion of context (previous words occurring in the sentence) helps the models to predict eye-tracking attributes better. this reaffirms previous observations made with language models that more context is always helpful. second, we find that xlm based models perform relatively better than the bert based models. third, our experiments show the advantages of augmenting additional linguistic features (word length and word frequency information in this case) to the contextual word representations to make better systems. this is in agreement with the findings from eye-tracking prediction tasks from last iterations of cmcl. fourth, we see how different pooling methods applied on the input token representations affect the final performance of the systems. finally, the experiments re-validate the approach of fine-tuning pretrained language models for specific tasks. hence we conclude that contextualized word representations from language models pretrained with many different languages, if carefully augmented, engineered, and fine-tuned, can predict eye-tracking features quite successfully.