MTL 782_ IITD at CMCL 2021 Shared Task: Prediction of Eye-Tracking Features Using BERT Embeddings and Linguistic Features (2021)

Abstract: Reading and comprehension are quintessentially cognitive tasks. Eye movement acts as a surrogate to understand which part of a sentence is critical to the process of comprehension. The aim of the shared task is to predict five eye-tracking features for a given word of the input sentence. We experimented with several models based on LGBM (Light Gradient Boosting Machine) Regression, ANN (Artificial Neural Network), and CNN (Convolutional Neural Network), using BERT embeddings and some combination of linguistic features. Our submission using CNN achieved an average MAE of 4.0639 and ranked 7th in the shared task. The average MAE was further lowered to 3.994 in post-task evaluation.

## introduction eye tracking data gauged during the process of natural and comprehensive reading can be an outset to understand which part of the sentence demands more attention. the main objective of the present experiment is to understand the factors responsible for determining how we perceive and process languages. the cmcl-2021 shared task @xcite focuses on predicting the eye-tracking metrics for a word. the goal of the task is to train a predictive model for five eye-tracking feature values namely, nfix (number of fixations), ffd (first fixation duration), trt (total reading time), gpt (go past time), and fixprop (fixation proportion) for a given word of a sentence @xcite . here, nfix is the total number of fixations on the current word, ffd is the duration of the first fixation on the prevailing word, trt is the sum of all fixation durations on the current word including regressions, gpt is the sum of all fixations prior to progressing to the right * * joint first author of the current word, including regressions to previous words that originated from the current word and fixprop is the proportion of the participants who fixated on the current word. with respect to eye-tracking data, regression refers to the backward movement of the eye required to reprocess the information in the text @xcite . in this work we have experimented with two broad categories of models: regessor based and neural networks based. among the regressor based models, we tried with catboost, xgboost, light gradient boosting machine (lgbm) among others. among the neural network based models we have used both ann and cnn. lgbm gave the best results among the regressor based models. cnn produced lowest mae between cnn and ann. in this paper we discuss the best models of each type and their corresponding parameters in detail. the paper is divided into the following sections: section 2 describes some details of the dataset used for the experiments. in section 3 we discuss the data preparation approaches for feature extraction. model details are presented in section 4, and section 5 presents analysis of the results. section 6 concludes the paper. the code for the proposed system is available at https: //github.com/shivaniiitd/eye_tracking ## dataset description the present task uses the eye-tracking data of the zurich cognitive language processing corpus (zuco 1.0 and zuco 2.0) @xcite . the dataset is divided into two subsets train, and test. the data statistics are presented in ## data pre-processing and feature selection it is important to identify the features that provide essential visual and cognitive cues about each word which in turn govern the corresponding various eye-tracking metrics for the word. in the present work we have used bert embeddings along with linguistic features @xcite to train the predictive models. mean absolute error (mae) was used for measuring the performance of the proposed systems for the shared task. before feature extraction, the following preprocessing steps were performed: ## proposed models in this section we present the details of the three predictive machine learning regression models namely, lgbm, ann and cnn. ## results and analysis the comparison among top four performing models, ranked according to mae, is presented in the lgbm models give the best results corresponding to nfix, ffd and fixprop among the top 4 best performing models, while cnn based model performed the best on avg_mae and gpt. as we observe in ## conclusion the aim of the present work is to develop a predictive model for five eye-tracking features. experiments were conducted using lgbm, ann and cnn models trained on a feature space consisting of pre-trained bert embeddings and linguistic features namely, number of syllables, pos tag, word length and word_id. the discussed cnn models achieved the best performance with respect to the test data. experiments for studying the impor-tance of individual features indicate that pos tag has the lowest impact on the overall mae, with respect to the cnn models and that the addition of syllables to the feature space in lgbm models does not improve the overall performance of the system. it is further observed that individual linguistic features lead to a varied effect on different eye-tracking metrics. separate tuning of hyperparameters and feature space corresponding to the lgbm and neural network based model, for each eye-tracking metric, can improve the overall system performance. even though cnn architecture is more complex, but with the same set of features the lgbm regressor gave almost same results. currently, we did not perform a rigorous hyperparameter tuning which may be taken up in future.