Poirot at CMCL 2022 Shared Task: Zero Shot Crosslingual Eye-Tracking Data Prediction using Multilingual Transformer Models (2022)

Abstract: Eye tracking data during reading is a useful source of information to understand the cognitive processes that take place during language comprehension processes. Different languages account for different cognitive triggers, however there seems to be some uniform indicatorsacross languages. In this paper, we describe our submission to the CMCL 2022 shared task on predicting human reading patterns for multi-lingual dataset. Our model uses text representations from transformers and some hand engineered features with a regression layer on top to predict statistical measures of mean and standard deviation for 2 main eye-tracking features. We train an end-to-end model to extract meaningful information from different languages and test our model on two separate datasets. We compare different transformer models andshow ablation studies affecting model performance. Our final submission ranked 4th place for SubTask-1 and 1st place for SubTask-2 forthe shared task.

## introduction eye tracking provides an accurate millisecond record of where people are looking while reading and is useful for descriptive study of language processing and understanding of the cognitive processing of brain related to reading. eye movements are many times language-specific because they depend on structure and ordering of words which are language dependent, however some features tend to be stable and universal and can be observed in all languages. modeling of human reading has been widely explored in psycholinguistics. the ability to accurately predict eye tracking between languages pushes this field forward, facilitating comparisons between models and analysis of their various functions. in this paper, we compare our eye-tracking prediction results with some simple baselines using token-level features, which we improve upon with our zero shot cross lingual model which we have described in section 3. ## our approach our models heavily use contextualised embeddings extracted from the pretrained models based on transformer architecture @xcite . we experiment on the training dataset with multilingual transformer models which are briefly described below : xlm @xcite is a transformer-based model that, like bert, is trained with the masked language modeling (mlm) objective. additionally, xlm is trained with a translation language modeling (tlm) objective in an attempt to force the model to learn similar representations for different languages. xlm-roberta @xcite ) uses self-supervised training techniques to achieve stateof-the-art performance in cross-lingual understanding. it is trained on unlabeled text in 100 languages extracted from commoncrawl datasets. these transformer methods use either wordpiece or bytepair model for tokenization, due to which we use only the first token embedding of the tokenized word by these methods. we use the above three tranformer models and attach the extracted output embeddings from these models to the manually constructed features which we have described in 3.1. the entire model architecture is explained in figure 1 . ## results and discussion ## conclusion and future work in this paper, we presented our approach to the cmcl 2022 shared task on eye-tracking data prediction. our models use the fusion model that involve using the multilingual contextualized token representations using transformer architecture and attaching input features that we created that aid the model performance in predicting eye tracking features. this approach helped us become language agnostic which essentially helped the model to perform well in the zero shot cross-lingual setting in subtask-2. our best model based on xlm-roberta outperforms the baseline and is also competitive with other systems submitted to the shared for both subtask-1 and subtask-2. although the embeddings from large language models as shown previously work fairly well as they consider the context of the sentence into consideration as well, possibly they can be improved further if take into consideration the surprisal index which would positively correlate with the reading time and fixation duration as shown in @xcite . in future, we aim to use more etymological features based on shared language history and also use the cross language lexical similarity index when predicting in cross lingual setting .