C og NLP - S heffield at CMCL 2021 Shared Task: Blending Cognitively Inspired Features with Transformer-based Language Models for Predicting Eye Tracking Patterns (2021)

Abstract: The CogNLP-Sheffield submissions to the CMCL 2021 Shared Task examine the value of a variety of cognitively and linguistically inspired features for predicting eye tracking patterns, as both standalone model inputs and as supplements to contextual word embeddings (XLNet). Surprisingly, the smaller pre-trained model (XLNet-base) outperforms the larger (XLNet-large), and despite evidence that multi-word expressions (MWEs) provide cognitive processing advantages, MWE features provide little benefit to either model.

## introduction and motivation many researchers now agree that eye movements during reading are not random @xcite ; as a result, eye-tracking has been used to study a variety of linguistic phenomena, such as language acquisition @xcite and language comprehension @xcite . readers do not study every word in a sentence exactly once, so following patterns of fixations (pauses with the eyes focused on a word for processing) and regressions (returning to a previous word) provides a relatively non-intrusive method for capturing subconscious elements of subjects' cognitive processes. recently, cognitive signals like eye-tracking data have been put to use in a variety of nlp tasks, such as pos-tagging @xcite , detecting multi-word expressions @xcite and regularising attention mechanisms @xcite : the majority of research utilising eye-tracking data has focused on its revealing linguistic qualities of the reading material and/or the cognitive processes involved in reading. the cmcl 2021 shared task of predicting human reading behaviour @xcite asks a * equal contribution slightly different question: given the reading material, is it possible to predict eye-tracking behaviour? our ability to quantitatively describe linguistic phenomena has greatly increased since the first feature-based models of reading behaviour (i.e. @xcite ). informed by these traditional models, our first model tests 'simple' features that are informed by up-to-date expert linguistic knowledge. in particular, we investigate information about multi-word expressions (mwes) as eye-tracking information has been used to detect mwes in context @xcite , and empirically mwes appear have processing advantages over non-formulaic language @xcite . our second model is motivated by evidence that pre-trained language models (plms) outperform feature based models in ways that do not correlate with identifiable cognitive processes @xcite . since many plms evolved from the study of human cognitive processes @xcite but now perform in ways that do not correlate with human cognition, we wished to investigate how merging cognitively inspired features with plms may impact predictive behaviour. we felt this was a particularly pertinent question given that plms have been shown to contain information about crucial features for predicting eye tracking patterns such as parts of speech @xcite and sentence length @xcite . we therefore had the goals of providing a competitive shared task entry, and investigating the following hypotheses: a) does linguistic/cognitive information that can be predicted by eye-tracking features prove useful for predicting eye-tracking features? b) can adding cognitively inspired features to a model based on plms improve performance in predicting eye tracking features? ## related work transformer architectures bidirectional encoder representations from transformers (bert) @xcite is a language representation model constructed from stacked neural network attention layers and 'massively' pre-trained on large natural language corpora. in contrast with traditional language models, bert is pre-trained in two settings: a 'cloze' task where a randomly masked word is to be predicted, and next sentence prediction. bert or derivative models have been used to achieve state-of-the-art baselines on many nlp tasks @xcite . analysis studies have shown that bert learns complex, task-appropriate, multi-stage pipelines for reasoning over natural language, although there is evidence of model bias. xlnet @xcite is an autoregressive formulation of bert which trains on all possible permutations of contextual words, and removes the assumption that predicted tokens are independent of each other. similar studies to our knowledge, studies that attempt to predict cognitive signals using language models are fairly few and far between. djokic et al. (2020) successfully used non-transformer word embeddings to decode brain activity recorded during literal and metaphorical sentence disambigua-tion. since rnns may be considered more 'cognitively plausible' than transformer based models, merkx and frank (2020) compared how well these two types of language models predict different measures of human reading behaviour, finding that the transformer models more accurately predicted self-paced reading times and eeg signals, but the rnns were superior for predicting eyetracking measures. in a slightly different task, sood et al. (2020) compared lstm, cnn, and xlnet attention weightings with human eye-tracking data on the movieqa task @xcite , finding significant evidence that lstms display similar patterns to humans when performing well. xlnet used a more accurate strategy for the task but was less similar to human reading. though these studies may indicate that transformer models are not the most suited to eyetracking prediction, they are still considered state of the art in creating broad semantic representations and general linguistic competence @xcite . as such, we hoped they would allow us to investigate carpenter and just's speculation that the dominance of word length and frequency for predicting eye-tracking behaviour may reduce "as the metrics improve for describing higher-level factors" like semantic meaning (1983, p. 290). ## experimental design 1 we pursued both feature engineering and deep learning approaches to the task; though both methods performed well independently, there was little improvement in predictive capability when combining their features (see ## results in the overall task. our overall standing is shown to be 5th, with an mae delta of 0.143 behind the best model. whilst a prediction which combined models 1 and 2 was slightly more accurate (see ## analysis and discussion our results ( we use normalised permutation feature importance (see appendix b) to better understand the value of different features and present it on a pertarget basis for each model in figure 2 . the most interesting outcome of our experiments was the fact that xlnet embeddings subsume information contained across most features except word length (especially in predicting nfix). it may be that the use of word-pieces obfuscate word length information thus requiring the explicit addition of that information. while the usefulness of features such as word length is consistent with the literature, we were surprised by the relative unimportance of mwe information given that many neurocognitive studies have demonstrated differences in how they are processed @xcite @xcite ). an additional surprise is that even though the skip-gram embeddings provide semantic information about single words as well as mwes, the feature rich models make little use of them. many of the feature rich models utilize the geco features, which may be because they provide approximate guidance about the distributions of the various gaze features that would be difficult to learn directly given the sparsity of the training data. ## conclusion and future work this work describes our submissions to the 2021 cmcl shared task: we contributed a feature rich model inspired by cognitive and linguistic information, and model predominantly based on contextual xlnet-base embeddings. we find that only a limited subset of the cognitive features (such as word length) are helpful in the xlnet model. to our surprise, neither xlnet-large embeddings nor mwe features provide performance improvements. however, we believe this indicates a need for further research into mwe representations as opposed to suggesting that mwes are unimportant for creating effective cognitive models.