Every word counts: A multilingual analysis of individual human alignment with model attention (2022)

Abstract: Human fixation patterns have been shown to correlate strongly with Transformer-based attention. Those correlation analyses are usually carried out without taking into account individual differences between participants and are mostly done on monolingual datasets making it difficult to generalise findings. In this paper, we analyse eye-tracking data from speakers of 13 different languages reading both in their native language (L1) and in English as language learners (L2). We find considerable differences between languages but also that individual reading behaviour such as skipping rate, total reading time and vocabulary knowledge (LexTALE) influence the alignment between humans and models to an extent that should be considered in future studies.

## introduction recent research has shown that relative importance metrics in neural language models correlate strongly with human attention, i.e., fixation durations extracted from eye-tracking recordings during reading @xcite @xcite @xcite . this approach serves as an interpretability tool and helps to quantify the cognitive plausibility of language models. however, what drives these correlations in terms of differences between individual readers has not been investigated. in this short paper, we approach this by analysing (i) differences in correlation between machine attention and human relative fixation duration across languages, (ii) differences within the same language across datasets, text domains and native speakers of different languages, (iii) differences between native speakers (l1) and second language learners (l2), (iv) the influence of syntactic properties such as part-of-speech tags, and (v) the influence of individual differences in demographics, i.e., age, vocabulary knowledge, depth of processing. taking into account individual and subgroup differences in future research, will encourage single-subject and cross-subject evaluation scenarios which will not only improve the generalization capabilities of ml models but also allow for adaptable and personalized technologies, including applications in language learning, reading development or assistive communication technology. additionally, understanding computational language models from the perspectives of different user groups can lead to increased fairness and transparency in nlp applications. ## method we analyse the spearman correlation coefficients between first layer attention in a multilingual language model and relative fixation durations extracted from a large multilingual eye-tracking cor-pus, including 13 languages @xcite as described below. total fixation time (trt) per word is divided by the sum over all trts in the respective sentence to compute relative fixation duration for individual participants, similar to @xcite . we extract first layer attention for each word from mbert foot_0 , xlm-r foot_1 and mt5 foot_2 , all three are multilingual pre-trained language models. we then average across heads. we also test gradient-based saliency and attention flow, which show similar correlations but require substantially higher computational cost. this is in line with findings in @xcite . eye-tracking data the l1 part of the meco corpus contains data from native speakers reading 12 short encyclopedic-style texts (89-120 sentences) in their own languages foot_3 (parallel texts and similar texts of the same topics in all languages), while the l2 part contains data from the same participants of different native languages reading 12 english texts (91 sentences, also encyclopedicstyle). for each part, the complete texts were shown on multiple lines on a single screen and the participants read naturally without any time limit. furthermore, language-specific lextale tests have been carried out for several languages in the l1 experiments and the english version for all participants in the l2 experiment. lextale is a fast and efficient test of vocabulary knowledge for medium to highly proficient speakers @xcite . for comparison, we also run the experiments on the geco corpus @xcite , which contains eye-tracking data from english and dutch native speakers reading an entire novel in their native language (l1, 4921/4285 sentences, respectively), as well as a part where the dutch speakers read english text (l2, 4521 sentences). the text was presented on the screen in paragraphs for natural unpaced reading. ## results in the following, we show results for the correlation analysis across languages and an in-depth analysis on different influences on those correlations. languages we compute the spearman correlation between relative fixation and first layer attention per sentence and average across sentences for all individual participants. we show correlation values averaged across participants for each language (l1) and corresponding data for english l2 in correlations for xlm-r are about 0.1 higher and for mt5 0.1 -0.2 lower compared to mbert. the correlation for english l2 are very similar between languages (0.3-0.34, mbert) and lowest for the english l1 participants (0.26, mbert). correlation values for geco are slightly lower for the dutch experiments but in the same range for the english part. processing depth to further analyse the different correlation values, particularly the low correlation in the l2 experiment for english native speakers, we look into skipping rates and total reading times and hereby focus on mbert to make results more comparable to @xcite . analyses on mt5 and xlm-r show similar results. figure 1 shows skipping rates and total reading times computed for individual participants on the entire dataset versus individual correlation values as computed above. we find significant correlations (@xmath0.01) for both skipping rate vs. correlation values (-0.41/ -0.34) and trt vs. correlation values (0.19/0.32) for l1 and l2 respectively. this indicates that more thorough reading, i.e., less skipping and more time per word, leads to higher correlation with first layer attention. we also see those correlations at language-level for some languages where he, fi, ru show highest scores at -0.7, -0.63, -0.59, respectively. for geco, we find similar trends for english (l1 and l2) but not for dutch. pos we look deeper into cross-lingual differences and show correlation values on token-level for 6 frequent pos tags in figure 2 . we extract relative fixations, standardise them to @xmath1. we use pos-tagging models from spacy and show results for the languages where respective models are available. 5 correlations for l1 are distributed similarly across different pos tags where adjectives show the highest correlation whereas verbs, although they carry an important part of the fixations, correlate much less. only korean poses an exception here where adjectives do not play the most prominent role in human attention and also correlate much less. here, nouns, pronouns, verbs and coordinating conjunctions correlate higher than in any other language and also much higher than adjectives. more research is required to interpret this finding. for l2, we see a very homogeneous distribution between languages and a similar distribution across pos tags as in most l1 experiments. lextale we show lextale scores for english l2 and fi, en, nl for l1 versus correlation values in figure 3 . we find a negative correlation for dutch speakers in l1 -0.36 and for the entire l2 data of -0.42 (@xmath2.05) suggesting that higher lextale scores lead to lower correlation with first layer attention. ## discussion & conclusion our results show that the correlation between relative fixation duration and first layer attention varies greatly across languages when read by native speakers. these differences can be attributed in part to the depth of processing: languages such as finnish and greek, which show high total reading times, show a more evenly distributed correlation pattern across the most frequent parts of speech. moreover, l1 english shows a high skipping rate and the lowest correlations. we find that more careful in-depth reading -processing more words for a longer time -correlates more strongly with attention than fast shallow reading. this is in line with previous research showing that attention patterns in bert carry high entropy values, i.e., are broadly distributed, particularly in the first layers @xcite , which also leads to higher correlation with fixation duration @xcite . the differences in skipping rate have various origins. on one hand, skipping rate is regulated by word length @xcite , which explains the lower skipping rate of agglutinative languages such as finnish and turkish @xcite , and in turn their higher correlation to mbert attention. on the other hand, word skipping is affected by l2 reading proficiency. more skilled learners make fewer fixations and skip more words @xcite . this is reinforced by our comparison between english l2 and native english reading (which shows lower correlation). this finding is also supported by our analysis on the lextale vocabulary test. lextale accurately estimates proficiency even at high levels @xcite . our results show that higher test scores lead to lower correlation with attention. again, this is due to the reading depth: highly proficient readers have a higher skipping rate @xcite . we furthermore looked at the influence of age and gender but could not find any meaningful differences. this might be due to the fact that all participants were university students, most of them under the age of 30, thus representing a very specific group of the overall population. it is also important to note that most of the languages in meco are indo-european and only 4 are not using the latin script. in summary, we have shown the impact of various subgroup characteristics reflected in reading and how they affect the correlation to neural attention. we argue that these differences should be taken into account when leveraging human language processing signals for nlp.