Team DMG at CMCL 2022 Shared Task: Transformer Adapters for the Multi- and Cross-Lingual Prediction of Human Reading Behavior (2022)

Abstract: In this paper, we present the details of our approaches that attained the second place in the shared task of the ACL 2022 Cognitive Modeling and Computational Linguistics Workshop. The shared task is focused on multi- and cross-lingual prediction of eye movement features in human reading behavior, which could provide valuable information regarding language processing. To this end, we train ‘adapters’ inserted into the layers of frozen transformer-based pretrained language models. We find that multilingual models equipped with adapters perform well in predicting eye-tracking features. Our results suggest that utilizing language- and task-specific adapters is beneficial and translating test sets into similar languages that exist in the training set could help with zero-shot transferability in the prediction of human reading behavior.

## introduction eye movements provide valuable information about the contents of underlying cognitive processes and where our attention falls @xcite . predicting human reading behavior as reflected in eye movements is an important task that requires capturing universal aspects of language processing as well as its language-specific properties @xcite . this task could help us gain insight into languagerelated eye movements and the predictive capabilities of the models of human reading behavior. various approaches have been proposed for the modeling of human reading behavior @xcite @xcite . the cmcl 2021 shared task focused on the prediction of 'monolingual' reading behavior and the participants applied various methodologies to predict eye-tracking features, e.g. gradient boosting, ensembling, using handcrafted features, deep learning @xcite @xcite @xcite . with regard to deep learning-based approaches, there exist findings suggesting that, as compared to transformer-based models @xcite , recurrent neural networks exhibit attention patterns closer to human attention @xcite . however, more recently, transformer-based models have been shown to better account for human reading behavior than recurrent neural networks @xcite . moreover, pretrained language models (plm) such as bert @xcite and xlm @xcite can predict multilingual human reading behavior well @xcite , in addition to having advanced the state-of-the-art in many downstream nlp tasks. the focus of the cmcl 2022 shared task @xcite is to predict four eye-tracking features for data containing sentences in 6 different languages as well as transferring to a new language. for this purpose, we train 'adapters' inserted into transformer layers of frozen plms @xcite . we find that training adapters for each language separately within multilingual transformers leads to good performance, attaining the second place in the leaderboard. in addition, we show that such models can transfer to new languages via simply translating the new test sets into closely-related languages (e.g. lexically or grammatically) that the model was exposed to during training. foot_0 2 background ## subtask 1: multi-lingual in this subtask, the aim is to predict eye-tracking features for data from 6 languages, for which we have training, validation and test sets. we focus on comparing a single setup for all languages vs. separate setups for different languages. ## subtask 2: cross-lingual for this subtask, we conduct various experiments to obtain results for the danish test set in the absence of training and validation data in this language. ## conclusion we have trained language-and task-specific adapters for the prediction of eye-tracking features reflecting human reading behavior in multi-and cross-lingual settings. our best models performed well, attaining the second place in the cmcl 2022 leaderboard. this suggests that pretrained language models enhanced with small adapter layers possess the capability to predict eye-tracking features. in addition to our setups, other methods such as dropping adapters or adapter fusion could be implemented @xcite . it would also be informative to consider autoregressive models and the possibility of making use of various lexical and syntactic features and additional cognitive signals. the prediction of each eye-tracking feature could also be informed by other eye-tracking features, as each of them represents different aspects of human reading behavior. similar approaches could also be of help in the modeling of other human cognitive signals, opening up novel ways of predicting and inspecting cognitive processes in humans.