Modeling Referential Gaze in Task-oriented Settings of Varying Referential Complexity (2022)

Abstract: Referential gaze is a fundamental phenomenon for psycholinguistics and human-human communication. However, modeling referential gaze for real-world scenarios, e.g. for task-oriented communication, is lacking the well-deserved attention from the NLP community. In this paper, we address this challenging issue by proposing a novel multimodal NLP task; namely predicting when the gaze is referential. We further investigate how to model referential gaze and transfer gaze features to adapt to unseen situated settings that target different referential complexities than the training environment. We train (i) a sequential attention-based LSTM model and (ii) a multivariate transformer encoder architecture to predict whether the gaze is on a referent object. The models are evaluated on the three complexity datasets. The results indicate that the gaze features can be transferred not only among various similar tasks and scenes but also across various complexity levels. Taking the referential complexity of a scene into account is important for successful target prediction using gaze parameters especially when there is not much data for fine-tuning.

## introduction in a situated interaction, interlocutors produce and interpret complex communicative signals and intertwine their verbal utterances with non-verbal signals like gaze. for instance, when referring to objects in the visual environment, speakers tend to fixate the target referent, listeners gaze at the objects they believe to be referred to by the speaker and, importantly, listeners monitor the speaker's gaze in case it provides reliable information about the referent @xcite . in noisy environments, listeners (that need to resolve references to objects) might even face situations where figure 1 : an example task-oriented scenario gaze provides a more reliable cue than spoken words. for example, in the scenario depicted in figure 1 , "the glasses" in the command illustrates an ambiguous reference and during human-human communication, such ambiguities can be easily resolved via tracking referential gaze. referential gaze as a communication modality is a well researched / fundamental phenomenon in psycholinguistics and cognitive science @xcite @xcite . mainstream natural language processing (nlp) systems -on the other handstill usually employ language-only approaches, where the performance is highly dependent on the completeness of the language modality. despite the fact that reference resolution in visual environments has become a very popular task in recent nlp and computer vision research @xcite @xcite @xcite @xcite , there is very little work on reference resolution models that exploit eye gaze for this task. they fall short on modeling referential gaze for realistic scenarios for task-oriented communication that go beyond simple settings. one main reason behind this is human gaze's intricate nature of being complex (a multivariate sequence) and multi-functional (e.g. referential gaze, social gaze and so on) @xcite . in this paper, we propose a novel task of predicting when the gaze is referential during the communication, aiming at modeling referential gaze for various multimodal settings. most recently, daily devices like laptops start to utilize eye-tracking technology @xcite @xcite . as a result, incorporating eye-movements in language comprehension models is an inevitable goal for nlp emerging from these developments, and this motivates systematic research on the interaction of different communicative modalities. however, the collection and pre-processing of eye-movement data is a very costly process, and this is another main reason why there are only a few large eyemovement datasets available @xcite @xcite . eye-movements are highly influenced by bottomup perceptual and top-down conceptual properties of the task (e.g. free viewing, search, etc.) and the properties of the visual environment @xcite @xcite . besides, their patterns (pupil size, saccade velocity, fixation duration, etc.) are very user-dependent @xcite . all these factors introduce challenges in (i) learning meaningful patterns from limited data, (ii) generalizing well enough to different kinds of situations of real-world complexities and (iii) successfully incorporating it to nlp systems for reference resolution and meaning recovery. to mitigate these problems, transfer learning can be used to adapt the knowledge obtained from one setting to another, benefiting from its added generalization capabilities. ## background in this paper, we apply transformer-based timeseries modeling and transfer learning to the phenomenon of referential gaze. section 2.1 discusses the background for technical modeling, and section 2.2 introduces referential gaze. ## approach we investigate the modeling of eye-movements and ask whether different referential complexities need individual referential gaze models or whether we can use transfer learning (pre-training on larger collections and fine-tuning on task-specific dataset). we build a model that predicts when the gaze of a participant is referential, i.e., when she looks at the target object referred to by the speaker. for a low-complexity scene (i.e., with few objects) and an unambiguous verbal description, this task can be considered as straightforward, since the user will quickly identify the target and not have to visually search for it. in a complex visual scene -with occluded objects and complex or ambiguous verbal descriptions -eye-movements can provide highly distinctive information to resolve ambiguities, but may also show more complex and challenging gaze patterns in return. therefore, in this study, we min-imize the contribution of accompanying linguistic and contextual information and focus on the influence and capabilities of gaze features. ## model architectures to establish the performance of within-and crosscomplexity performances, we employ two deep learning approaches; (i) lstm as a sequential base model and (ii) transformer architecture @xcite . we use a transfer learning approach to establish the compatibility of different complexities. this is done because there are not many large datasets available and we want to study options of how an available benchmark (eye4ref) can be utilized as a baseline that is then adapted on a small set of individual, task-specific data. transfer learning only trains the final layer (the output layer and all dense layers are frozen), thus the input representation stays the same. therefore, we further experiment with fine-tuning the layers to arrive at an input encoding that better fits the small target data. the full code, and model summaries are provided under supplementary material. baseline lstm model we experimented with two variations of a bi-directional lstm architecture @xcite . since we are dealing with a sequence classification task, attention mechanisms can help to improve the performance of our model by guiding the model to give more weight to the relevant time-frames in the sequence. in the second variation, we use a variant of self-attention @xcite known as the sequential self attention by keras. the details of the models are provided in appendix a.3. ## conclusion attending to referential gaze of the interlocutors is fundamental to face-to-face communication, yet still mostly ignored by the nlp community. in this study, we experiment with two deep learning methods (lstm and transformer) to model referential gaze. we target gaze-only reference resolution and test how we can transfer the gaze features among various scene settings. depending on the task (target or all-referent prediction) and the complexity level, the models exhibit different advantages. while tst is successful at generalizing from low complexities to higher ones and without the need of extra fine-tuning step, lstm beats tst at generalizing from the medium conditions. but its performance is positively affected by fine-tuning. one of the challenges of eye-movement modeling originates from being highly individual, task and environment dependent, making the generalization is more challenging. the results on different levels of gaze parameter normalization indicate that long-tail conditions clearly benefit from using more globally normalization. within-complexity comparisons show that gaze features based on one scenario can be useful for similar new scenes. however, adopting among various complexities using pretrained models (with or without fine-tuning) displays encouraging results. yet these result also confirm the challenging nature of the task and provide stepping stone for modeling referential gaze. especially, the results are not trivial considering that we only use low-level gaze features. in addition to the gaze parameters, including the number of objects in the scene as a feature improves referential gaze prediction, indicating that this information makes the model more sensitive to various referential complexities.