Eyes on the Parse: Using Gaze Features in Syntactic Parsing (2020)

Abstract: In this paper, we explore the potential benefits of leveraging eye-tracking information for dependency parsing on the English part of the Dundee corpus. To achieve this, we cast dependency parsing as a sequence labelling task and then augment the neural model for sequence labelling with eye-tracking features. We also augment a graph-based parser with eye-tracking features and parse the Dundee Corpus to corroborate our findings from the sequence labelling parser. We then experiment with a variety of parser setups ranging from parsing with all features to a delexicalized parser. Our experiments show that for a parser with all features, although the improvements are positive for the LAS score they are not significant whereas our delexicalized parser significantly outperforms the baseline we established. We also analyze the contribution of various eye-tracking features towards the different parser setups and find that eye-tracking features contain information which is complementary in nature, thus implying that augmenting the parser with various gaze features grouped together provides better performance than any individual gaze feature.

## introduction when it comes to natural language processing (nlp), most of the research work that is done is based on textual data that is usually annotated in some way while human language processing data like eye movement recordings, etc. are often overlooked. the solutions for most tasks in nlp these days are based on neural networks which require annotated data on a large scale to learn good representations and provide meaningful results. annotating such large quantities of data is not an easy task usually requiring a lot of manual labour which is time consuming and can be expensive not to mention resolving the inter-annotator agreement. however, every single day, millions of people read their daily newspapers, books, magazines, articles on the internet, etc. in a multitude of languages. in the past couple of years some studies have shown that using behavioral data for nlp tasks involving syntax and semantics can be useful @xcite @xcite @xcite . this leads us to believe that if there was a way to tap into cognitive data generated by unconscious human parsing of text, it could be leveraged in some manner to support nlp tools thereby reducing our dependency on annotated textual data. another reason for choosing to work with eye-tracking features is that it is highly likely that eyetracking technology will be available on a much larger scale in the near future and hence can be leveraged easily for nlp tasks. this is evidenced by the availability of eye-tracking through regular webcams @xcite and smartphones @xcite . @xcite developed the itu gaze tracker which was a low-cost eye-tracking system based on a webcam that is mounted close to the user's eye. the developers of this system soon established their company "the eye tribe" 1 with a focus on providing low cost eye-tracking and providing eye control technology for mass market consumer devices. although eye-tracking recording might seem an even more expensive solution to manually annotating data, several studies and even efforts made by the industry indicate that costs associated with eye-tracking hardware and software are just going to go down and that next generation mass consumer goods like smartphones, tablets, laptops and gaming consoles will come equipped with eye-tracking capabilities. dependency parsing is a fairly well studied phenomenon because of its relevance in several downstream nlp applications and because of the overwhelming success of the universal dependencies (ud) project @xcite which is an endeavor to provide guidelines for consistent dependency annotations across multiple languages. this study primarily presents an investigation in leveraging eye-tracking features for dependency parsing and the knowledge they bring in identifying syntactic categories and relations. there have been only two previous studies, to the best of our knowledge, which try to leverage gaze features foot_0 for dependency parsing @xcite . @xcite cast dependency parsing as sequence labelling in a multi-task learning (mtl) setup where the gaze features are predicted as an auxiliary task while considering that the eye-tracking data will be available only during training time. for our study, we try an alternative approach to @xcite wherein we focus on determining the effects of incorporating eye-tracking data directly in the model for dependency parsing and assume that eye-tracking data would also be available to us at inference time as opposed to the approach followed by @xcite . to provide further evidence, we also employ a graph-based parser and augment it with gaze features to see if they improve parsing. we try to address the following research questions through our study: ## parser descriptions in this section, we describe the two parsers we use in our work. our primary parser is a modification of the system created by @xcite who in turn adapted the ncrf++ system @xcite which is an open source neural sequence labelling toolkit. our secondary parser is a more traditional graph-based parser known as the bist parser from @xcite . one of the main differences between our parsers and the parser by @xcite is that we assume the availability of gaze features both at training and test time whereas @xcite only use the gaze features during training. another difference is that we incorporate gaze features as the input to the parser whereas @xcite predict the gaze features as an auxiliary task in a multi-task learning setup. the parsers are evaluated with respect to the labelled attachment score (las) and the unlabelled attachment score (uas). the las is concerned with the number of words that are assigned a correct head as well as the correct dependency relation whereas the uas is just concerned with the words that are assigned a correct head. ## using gaze features for dependency parsing in this section, we describe the dataset that we use along with a description of the gaze features that we intend to make use of. we then describe a set of experiments with these gaze features and report the unlabelled and labelled attachment scores (uas and las) on the development and test set and analyze the result of all the scenarios. ## results the results from the dev set for the lexical sequence labelling parser show that the grouped context gaze features provide the best improvements as compared to the baseline with an improvement of +0.99 for uas score and +0.73 for the las score. individually, first fixation duration provides almost the same improvements as the grouped context features with an improvement of +0.97 for uas score and +0.73 for las score. these results are in line with the results of @xcite , who found improvements with the early and context grouped gaze features. interestingly, the results from the dev set show that in case of delexicalized parsing, the set of basic gaze features, grouped together provide the best improvements as compared to the baseline. they provide an improvement of +0.71 for the uas score and +0.42 for the las score. individually, the w+2 fixation duration and n fixations features seem to provide the best improvements over the baseline. a further analysis of the output 6 of the delexicalized sequence labelling parser shows that the dependency relations towards whose identification the gaze features help the most are root, cop and aux. the relative improvements in the prediction of these relations are 9%, 13% and 15% respectively. the pos tags for which gaze features help in correctly identifying the head and dependency relation the most are nouns, conj and verbs. the high relative improvements for root, aux, cop and verb makes us conclude that gaze features contain some important syntactic information related to verbs allowing the parser to parse the verbs much better. as verbs are heads of syntactic clauses, it may also be that gaze features help the parser distinguish the main clause (headed by a verb with the 'root' label) from subordinate clauses. the improvement for conj is also high meaning gaze features somehow help the parser in understanding co-ordination structures. an alternate hypothesis to explain the improvements in delexicalized parsing could also be that since our data makes use of google universal pos tags @xcite , which is more coarse than the ud tagset, the gaze features may simply be distinguishing the pos categories into more fine grained ones like distinguishing between full verbs and auxiliary verbs or co-ordinating and sub-ordinating conjunctions which then in turn helps the parser. it might be the case that were the ud pos tagset had been used in the data, the improvements in the delexicalized parser might have been smaller. ## conclusion in this paper, we explored the benefits of using eye-tracking features for dependency parsing. we performed a set of experiments wherein we tried different parser settings and different eye-tracking feature selection along with various feature normalization techniques to try to answer the question of relevance of gaze features for syntactic parsing. our experiments show that although eye-tracking features seem to help in dependency parsing where all features (word level, character level and pos tags) are used for the las score, the improvements over the baseline are not statistically significant. however, when we use eye-tracking features for delexicalized parsing, our experimental results show that there is statistically significant improvement over the baseline for both the uas and las scores. for this particular setup, our results seem to indicate that using a mixture of raw and normalized eye-tracking features seems to provide the best improvements. the results also seem to indicate that grouped gaze features perform better than individual gaze features and that only on combining different gaze feature groups can we obtain significant improvements over the baseline if any. this seems to suggest that the various eye-tracking features contain information which is complementary in nature. our experiments also seem to show that gaze features contain syntactic information allowing the parser to parse verbs better and also to better understand co-ordination structures.