TALEP at CMCL 2021 Shared Task: Non Linear Combination of Low and High-Level Features for Predicting Eye-Tracking Data (2021)

Abstract: In this paper we describe our contribution to the CMCL 2021 Shared Task, which consists in predicting 5 different eye tracking variables from English tokenized text. Our approach is based on a neural network that combines both raw textual features we extracted from the text and parser-based features that include linguistic predictions (e.g. part of speech) and complexity metrics (e.g., entropy of parsing). We found that both the features we considered as well as the architecture of the neural model that combined these features played a role in the overall performance. Our system achieved relatively high accuracy on the test data of the challenge and was ranked 2nd out of 13 competing teams and a total of 30 submissions.

## introduction the cmcl 2021 shared task @xcite aims at comparing different approaches to the task of predicting eye tracking variables. given english text as a sequence of tokens, the goal is to predict 5 behavioural metrics (averaged over 12 human subjects). our approach was based on two major steps. first, we generated several features that either proved successful in previous work or that reflected our intuition about their potential in predicting eye tracking data. second, we proposed a way for an optimal combination of these features using a simple neural network. both steps proved to be useful in our final predictions. the paper is organized as follows. first, in section 2, we list and describe the features we used to predict the eye-tracking data. then, in section 3, we briefly describe the neural architecture of the model we used to combine the features. next, in section 4, we report details about the model's training and evaluation. finally, in section 5, we analyze and discuss the impact of each feature (as well as the impact of each group of features) on the overall performance of the model. ## features generation before introducing the model (schematically described in figure 1 ), we thought useful to first list and describe how we obtained the candidate predictive features from the original textual material of the challenge as well as from secondary sources. our features are listed in ## features combination we used a simple neural network to perform a series of five regression tasks, one for each measure to predict. the regressions were realized using the architecture described in figure 1 . as is clear from the figure, all regression tasks take as input a single representation of the words being processed. this representation is optimized for the five regression tasks simultaneously. perhaps the most complex part of the model is the encoding of the input layer because different kind of features require different encoding methods. 2 the model is composed of three parts, each part taking as input the output of the preceding one (figure 1 ). ## training and evaluation in this section, we will describe how we trained the neural network presented in section 3 to predict all five shared task oculometric measures (nfix, ffd, trt, gpt, fixprop), and how we proceeded for the choice of its hyperparameters such as number of layers, size of layers, size of embeddings. during the training phase of the shared task, we decided to split the training material into train/dev/test parts of the following respective sizes 70%/15%/15%. this allowed us to use the dev part for early stopping and the test part to compare competing models on the basis of their generalization capability. we used absolute error as our loss function, and used adagrad as our optimizer. to decide on the values of the hyperparameters, we trained different models (changing one hyper parameter at a time) for 40 epochs on the train part of our split. as a form of early stopping, we only saved a model when its performance was the best on the dev set. finally, we used performance on the test part of our split to compare models and decide which hyperparameter values were the best. to train our final model, we ditched our custom split and used the entire shared task training material for a total of 7 epochs to avoid overfitting, achieving an mae of 3.83 (the best team obtained 3.81). in ## results and discussion the individual mae values were computed by training the model only on the feature at hand in addition to frequency and length, thus reflecting its performance beyond these simple baseline features foot_2 . as for the group-level mae, we obtained them by training a model that takes as input all the features of the group at hand as well as the preceding groups in the table. for example, the mae for "raw textual features" was obtained by training the model on all the feature in this group only (as there is no preceding group). the mae for "frequencies" was obtained by training the model on all the feature in this group in addition to the features in the "raw textual features" group, and so forth. finally, the score associated with "complexity metrics" is the most comprehensive, including all features in the table. the goal of such nested calculation is to appreciate the role of each higherlevel group above and beyond the information provided by the lower-level group of features. the four groups were ordered by the amount of effort it requires to obtain them. we did not test every combination of features. several conclusion can be drawn from these results. first, we found that low-level features performed very well. indeed when combining only raw textual features and frequencies, we already had an impressive performance of @xmath0.78. the linguistic features allow us to only slightly improve performance with a small gain of ∆@xmath1.04. surprisingly enough, the complexity metrics barely added any useful information. when looking at each complexity measure individually, we found that only the measures related to the stack size of the parser added information, whereas entropy-based measures, if anything, degraded performance in the test set compared to frequency and length. this was unexpected because the literature @xcite suggest that these metrics should play a little but noticeable role in modeling oculometric features. we suspect that even if these metrics are significant in mixed effect models, they are not powerful enough to increase the predictive performance of a neural network model. in addition to testing the contribution of the predictive features, we were curious if our way of combining these features also played a role. thus, we compared our non-linear neural network to five linear regressions (one for each variable in the challenge) both using