G aze VQA : A Video Question Answering Dataset for Multiview Eye-Gaze Task-Oriented Collaborations (2023)

Abstract: The usage of exocentric and egocentric videos in Video Question Answering (VQA) is a new endeavor in human-robot interaction and collaboration studies. Particularly for egocentric videos, one may leverage eye-gaze information to understand human intentions during the task. In this paper, we build a novel task-oriented VQA dataset, called GazeVQA, for collaborative tasks where gaze information is captured during the task process. GazeVQA is designed with a novel QA format that covers thirteen different reasoning types to capture multiple aspects of task information and user intent. For each participant, GazeVQA consists of more than 1,100 textual questions and more than 500 labeled images that were annotated with the assistance of the Segment Anything Model. In total, 2,967 video clips, 12,491 labeled images, and 25,040 questions from 22 participants were included in the dataset. Additionally, inspired by the assisting models and common ground theory for industrial task collaboration, we propose a new AI model called AssistGaze that is designed to answer the questions with three different answer types, namely textual, image, and video. AssistGaze can effectively ground the perceptual input into semantic information while reducing ambiguities. We conduct comprehensive experiments to demonstrate the challenges of GazeVQA and the effectiveness of AssistGaze.

## introduction computer vision systems for understanding data, such as video clips, photos, and generated synthetic images, have been studied for a few decades and have achieved remarkable progress @xcite @xcite @xcite @xcite . many projects @xcite @xcite @xcite @xcite in recent years have shown promising developments in the usage of multimodal information. moreover, there are several vqa studies, such as tutorialvqa @xcite , agqa @xcite , tvqa @xcite , knowit vqa @xcite , egovqa @xcite , ego-taskqa @xcite , gqa @xcite , dramaqa @xcite , vqa-mhug @xcite , pointqa @xcite ) that target at different aspects such as instructional assistance, subtitle inclusion, and compositional reasoning. despite the notable progress in vqa studies, most of the works use videos from one or multiple third-person views (tpv). constrained by the fixed viewing angle, tpv video may not be able to capture subtle information on human operation in industrial tasks such as assembly and disassembly. in particular, the tpv may not be sufficient to capture the interaction that forms the basis of human-human/robot shared tasks. first-person view (fvp) videos containing gaze information are beneficial to understanding the tasks performed in assembly-disassembly studies and overcoming spatial ambiguities. a novel task-oriented collaborative vqa is proposed in this research based on the collaboration theory, which is defined as a coordinated, synchronous activity resulting from a continuous effort to construct and maintain a shared understanding of a problem @xcite . this theory encompasses four key aspects: situation, interaction, process (grounding), and effect @xcite . situation is characterized by a degree of division of labor, which sets the stage for interactions. interaction is characterized by cognitive process such as grounding and mutual modeling, leading to effect that contribute to shared understanding @xcite . the ultimate goal of collaboration, according to this theory, is to build the common grounds necessary for effective cooperation. in the context of interaction between two subjects, there is a requirement to develop a shared language for task completion. this concept is called grounding mechanism @xcite , which operates on a functional principle: while perfect mutual understanding is not attainable, nor necessary, it is required to sustain sufficient mutual understanding to continue the task @xcite . in addition, every collaboration has a unique user experience. however, the tpv fpv videos may not adequately capture the semantic meaning that represents users' intentions and unique experiences. the gazevqa task aims to establish common ground for collaborative tasks involving humanrobot interaction. furthermore, different from other existing vqa studies, gazevqa leverages common grounding theory @xcite and uses eye-gaze information to facilitate semantic understanding in collaboration tasks. as shown in figure 1 , gazevqa contains gaze-augmented qa pairs that are related to human-human interaction, which can assist a person in learning task knowledge with an ai assistant that answers questions. to tackle the challenges of visual understanding presented in gazevqa, we propose assistgaze -an assistive model for task-collaborative hri-qa. it is a combination of different models to answer the questions with three different answer types, including textual answers, video retrieval answers, and image answers. the main purpose of assistgaze is to predict correct answers for task-specific questions in gazevqa by using gaze information. we inquire into different configurations of the proposed model, such as availability of gaze, choice of encoders, answer types, etc., and compare our model with a few benchmarks, such as random guess and violet @xcite . first, random guess models and masked versions of all video features are examined. secondly, the effectiveness of tpv videos is examined. finally, fpv videos with gaze information are added to the experiments. it is observed that the model obtains the highest performance when all features are included. there are three contributions of this paper. (1) we propose a novel collaborative vqa task to support collaboration in industrial applications such as the assembly and disassembly processes. (2) we design gazevqa, a new vqa dataset on collaborative scenarios, with carefully curated qa pairs and multimodal inputs including videos captured from multiviews and eye gaze information in the fpv video. (3) we create a new assistant model, assistgaze, to generate answers in three formats, while using gaze information to resolve spatial ambiguities. the model assists a human user in collaboration and interaction by answering task-specific questions. ## conclusions in this paper, with a new dataset gazevqa, we propose a novel approach to task-oriented collaborative qa towards real-world applications like human-human/robot collaboration. the gazevqa depicts the natural qa collaboration interaction between an instructor and a subject during guided assembly and disassembly tasks. moreover, we propose a new method with a multiple-choice answer format for all types of questions designed to perpetuate the common grounding theory for collaboration tasks. unlike the legacy vqa studies, we address a challenging task in the industrial settings on collaborative task execution. the proposed assistgaze is a new baseline for evaluating the vqa model's ability to address this challenging task. in future studies, further developments are required to improve the collaboration of computer vision with semantic understanding to inquire into the collaboration of assistgaze and gazevqa on realworld collaborative hri applications. more efforts are needed to address high-level challenges such as temporal action segmentation, action recognition, object recognition, and spatial understanding tasks. we hope that the proposed novel gaze-enhanced vqa dataset "gazevqa" and the new assistant model "assistgaze" will enable the community to move forward with task-oriented vqa.