NU HLT at CMCL 2022 Shared Task: Multilingual and Crosslingual Prediction of Human Reading Behavior in Universal Language Space (2022)

Abstract: In this paper, we present a unified model that works for both multilingual and crosslingual prediction of reading times of words in various languages. The secret behind the success of this model is in the preprocessing step where all words are transformed to their universal language representation via the International Phonetic Alphabet (IPA). To the best of our knowledge, this is the first study to favorably exploit this phonological property of language for the two tasks. Various feature types were extracted covering basic frequencies, n-grams, information theoretic, and psycholinguistically-motivated predictors for model training. A finetuned Random Forest model obtained best performance for both tasks with 3.8031 and 3.9065 MAE scores for mean first fixation duration (FFDAvg) and mean total reading time (TRTAvg) respectively.

## introduction eye movement data has been one of the most used and most important resource that has pushed various interdisciplinary fields such as development studies, literacy, computer vision, and natural language processing research into greater heights. in a technical point of view, correctly determining theoretically grounded and cognitively plausible predictors of eye movement will allow opportunities to make computational systems leveraging on these properties to be more human-like @xcite . common human reading prediction works make use of the standard latin alphabet as it is internationally used. however, investigating eye movement and reading patterns in other non-anglocentric writing systems such as chinese and bengali is as equally as important @xcite . fortunately, there is a growing number of previous works exploring multilinguality in eye tracking prediction both in data collection and novel prediction approaches. the study of @xcite was the first to explore potential crosslinguality of chinese, english and finnish which differ in aspects of visual density, spacing, and orthography to name a few. the results of the study favorably support possible universality of representation in reading. in the same vein, @xcite was the first to try use of large finetuned multilingual language models like bert @xcite and xlm @xcite in a crosslingual setting to predict eye tracking features across english, dutch, german, and russian. data-wise, the published works of @xcite for meco, @xcite for the dundee corpus, and @xcite for geco have made significant impact in the field where they covered curation and collection of eye-tracking corpus for other languages in addition to english. ## task definition and data the cmcl 2022 shared task @xcite foot_1 describes two challenges: predicting eyetracking features in a multilingual and crosslingual setup. the eye movement dataset for this shared task contains sentences written in six languages: mandarin chinese @xcite , hindi @xcite , russian @xcite , english @xcite @xcite , dutch @xcite , and german @xcite . the mean first fixation duration (ffdavg) and mean total reading time (trtavg) as well as their corresponding standard deviations (ffdstd and trtstd) are the four main eye-tracking features that need to be predicted by the participants through proposed computational means. for the multilingual task, the training, validation, and testing datasets conform to the identified six languages. while for the crosslingual task, a surprise language (danish) is provided as the test dataset. ## eye-tracking prediction in universal language space the proposed solution in this work is inspired by both classical and recent previous works in speech recognition systems @xcite @xcite with multilingual and crosslingual capabilities through the transformation of words or similar sounding units in one global shared space using the international phonetic alphabet (ipa). this functionality allows models to generalize and adapt parameters to new languages while maintaining a stable vocabulary size for character representation. by definition, the ipa contains 107 characters for consonants and vowels, 31 for diacritics for modifying said consonants and vowels, and 17 signs to emphasize suprasegmental properties of phonemes such as stress and intonation @xcite . figure 1 describes the unified methodology used for tackling both the multilinguality and crosslinguality challenge of the shared task. the backbone of this proposed solution lies with the phonetic transcription preprocessing step to convert the raw terms from the data written in mandarin chinese, hindi, russian, english, dutch, and german to their ipa form. we used epitran by @xcite for this process. the surprise language for the crosslingual task, danish, is not currently supported by epitran. we instead resorted to use automatic phonetic transcriber 3 , a paid transcription service that caters the danish language. the transcription cost of the danish test data is c15. ## results ## conclusion precise eye movement datasets in multiple languages are considered one of the most important contributions that benefit various interdisciplinary fields such as psycholinguistics, developmental studies, behavioral studies, computer vision, and natural language processing. in this paper, we present a novel method of transforming multilingual eye-tracking data (english, mandarin, hindi, russian, german, dutch, and danish) to their ipa equivalent, enforcing a single vocabulary space which allows competitive results for both multilingual and crosslingual tasks in a regression analysis setup. future directions of this paper can explore more cognitively and theoretically plausible features that can be extracted as well as deeper interpretation studies of the predictive models trained.