Situation-Specific Multimodal Feature Adaptation (2021)

Abstract: In the next decade, we will see a considerable need for NLP models for situated settings where diversity of situations and also different modalities including eye-movements should be taken into account in order to grasp the intention of the user. However, language comprehension in situated settings can not be handled in isolation, where different multimodal cues are inherently present and essential parts of the situations. In this research proposal, we aim to quantify the influence of each modality in interaction with various referential complexities. We propose to encode the referential complexity of the situated settings in the embeddings during pre-training to implicitly guide the model to the most plausible situation-specific deviations. We summarize the challenges of intention extraction and propose a methodological approach to investigate a situation-specific feature adaptation to improve crossmodal mapping and meaning recovery from noisy communication settings.

## motivation in recent years, we have witnessed a considerable increase in the use of assistive technologies that can engage in communication and perform tasks. these can come in different forms like smart speakers and mobile devices that you can command with audio, or more specialized task-oriented robots that can actually realize users' command in 3d environments. the steady increase in the use of collaborative robots @xcite in daily life brings along another important human-computer interaction theme: the capability of engaging in a natural and smooth spoken dialog with humans, which is a major scientific and technological challenge. particularly, being able to follow a communication that conveys thoughts and intentions expressed in a flexible manner without the restrictions of a close-set of commands is a crucial component of assistive robots for the handicapped and elderly people and for the education / entertainment purposes. spoken and situated communication is composed of various perceptual (e.g. audio, visual) and representational modalities (e.g. language, deictic eye-movements, gestures). effectiveness and fluency of human communication capabilities inspire us to develop robust language models that can deal with uncertainties by evaluating all the available information from multiple sources and reach a good-enough decision. in order to reach this performance, we need to model our situated language understanding systems to incorporate those modalities and let them interact in a meaningful way. this brings forth some important questions; how to integrate different modalities and how to utilize adaptive processing for effective situationawareness to be able to deal with cases where some of the modalities are restricted due to noise in the communication channels. this capability for crossmodal integration can be a very important feature in resolving references or executing commands for smart speakers or helper robots that aid people in their daily activities. ## situated language understanding in a task-oriented setting (e.g. helper robots completing a given task), the goal of natural language communication is to extract the intention of the speaker. such communication usually happens in structurally rich visual environments like the one in figure 1 , which contains several glasses of different types (wine and tea glasses), mice (computer mouse and cat toy), windows (open and closed) etc.. some of them are even (partially) occluded from the viewer's perspective. the environments usually also include people and their interactions with the objects (actions). visual information plays a crucial role in determining the referential objects related with the action and to accomplish the task. thus, computational solutions that incorporate those cues are expected to perform better in grasping the meaning compared to their (text-only) unimodal counterparts (see alac ¸am et al. (2020a) for a review). determining the correct intention of a user is not always straightforward due to various reasons. let us take the following sentences as an example: in both cases, on the syntactic level, the pronoun it refers to the wine. but in the referential world, the first one clearly refers to a wine bottle, while the second (with a lesser degree of certainty) refers to a glass of wine. in many cases, even the object names are verbally omitted in spoken utterances. this kind of implicit commands like "i prefer red, can you open a bottle and bring it to me?", require the hearer to reconstruct the underlying intention "open a bottle of red wine, and bring the bottle" @xcite . alternatively, depending on the spatial arrangements of the agents and the objects in the room, the intention of the speaker might be slightly different and more complex. for example, when the empty glasses are closer to the listener than the speaker, the interpretation might be: "open a bottle of red wine, pour the wine into one of the empty wine glasses and bring the glass of wine to me". expressing this intention explicitly most often results in unwieldy utterances. furthermore, when the environment is noisy, or the communication partner suffers from a motor or cognitive impairment, multimodal integration plays a more critical role. noise in communication can originate from various sources. it can be linguistic noise (e.g. spelling mistakes, complex attachments), visual ambiguities (e.g. clutter in the environment, occlusions) or an acoustic noise. instead of waiting for clarification, combining the uncertain information from the linguistic channel with information from the other ones increases the fluency and the effectiveness of the communication @xcite . one of the most well-known examples to this phenomenon is the cocktail party effect, that highlights the human ability to focus on one particular source while inhibiting the noisy ones. when the informativeness of one modality is reduced due to environmental conditions, the human language processing system can successfully adjust itself by relying less on the unclear modality and using other cues in the environment. in this specific scenario, other informative cues provide more reliable information compared to the noisy linguistic input. these cues can come from the surrounding environment and from the communicational partners, and include eye-gaze direction or representational gestures combined with their referential link to the entities in the environment. eye-tracking is attracting considerable interest in many assistive technologies such as educational vr systems that provide embodied learning environments or driver monitoring systems. the use of eye-tracking in daily technological products such as mobile phones, laptops and virtual reality headsets is increasing day by day @xcite @xcite . therefore, incorporating eye-movements in our language comprehension models is an inevitable outcome of these latest developments, and this makes the systematic research on the combination of this modality with others very crucial. ## dataset collection the success of a situation-aware language comprehension model is highly dependent on the representativeness of the modalities under various conditions. this kind of coverage requires a richly annotated multimodal corpus that displays the variety of language expressiveness and flexibility. developing such a corpus is a very costly process. thus, a dataset that profoundly incorporates a variety of modalities and their various aspects addressing language comprehension tasks is currently not available. therefore, we plan to train the model on a set of available multimodal datasets (as listed below), by using whichever modality constellations they can offer (including datasets from various domains like psycholinguistics, language technology, computer vision, human-robot interaction etc.) in a stepwise manner; namely starting with simple / few relations, then gradually increasing the complexity of the interactions. there are general-purpose multimodal datasets that can be used for training: ## objectives one of the main objectives of this research proposal is to quantify the effect of each modality and their interactions by conducting systematic empirical research with computational modeling and human subject studies. another objective lies in creating multimodal and multilayer embedding spaces in which the layers will be sensitive to various situation complexities, an approach that has not been considered yet. moreover, eye-movements of the speakers, as a substantial but underrepresented component of face-to-face communication, are incorporated to further improve nlp methods on meaning extraction and crossmodal reference resolution. ## conclusion in this research proposal, we focus on three factors that can enhance the communication between humans and assistive technologies. the first one is the encoding of the referential complexity of the situated settings while creating multimodal embeddings. as pointed out in @xcite , pre-trained models, that were created by fusing the modalities without constraints, are expected to be an out-of-the-box solution and work well for a variety of simpler tasks. in this research, we propose to encode referential complexity during the training phase to see whether the complexity-sensitive embeddings will improve the tasks of crossmodal mapping and meaning recovery. we believe that this will implicitly direct the model to focus on various textual and visual forms of the same concepts. the second factor is the inclusion of eyemovements as an additional modality to enhance meaning recovery from noisy settings where some parts of the sentences or visual labels are masked. at last, this research will also contribute to a better understanding of the contributions of each individual modality, of amodal and modality-specific features and their interactions. the proposed method will be beneficial for other task-oriented communication scenarios, where the cognitive systems need to understand the intention and to aid the user in the most efficient and effective way, such as educational video-games, training simulations, and assistive navigation systems.