A Visually-grounded First-person Dialogue Dataset with Verbal and Non-verbal Responses (2020)

Abstract: In real-world dialogue, first-person visual information about where the other speakers are and what they are paying attention to is crucial to understand their intentions. Non-verbal responses also play an important role in social interactions. In this paper, we propose a visually-grounded first-person dialogue (VFD) dataset with verbal and non-verbal responses. The VFD dataset provides manually annotated (1) first-person images of agents, (2) utterances of human speakers, (3) eye-gaze locations of the speakers, and (4) the agents’ verbal and non-verbal responses. We present experimental results obtained using the proposed VFD dataset and recent neural network models (e.g., BERT, ResNet). The results demonstrate that first-person vision helps neural network models correctly understand human intentions, and the production of non-verbal responses is a challenging task like that of verbal responses. Our dataset is publicly available.

## introduction in recent years, visually-grounded dialogue systems have attracted increasing attention @xcite @xcite . for example, @xcite developed an image-grounded conversational agent that uses visual sentiment, facial expression, and scene features, and @xcite constructed the publicly available igc dataset, which comprises image-grounded conversations. although these studies and resources have been shown to be useful, there are currently two limitations. first, in image-grounded dialogue tasks, figure 1 : example of proposed vfd dataset. "u", "v", and "n" denote a human utterance, the agent's verbal response, and the agent's non-verbal response (i.e., action), respectively. all utterances and responses are represented in japanese. english translations are added below for easier understanding. the red line links the eyes to the gaze location. another important limitation is that, although dataset type perspective response size visdial @xcite task oriented third-person verbal 120k mmd @xcite task oriented third-person verbal 150k talkthewalk @xcite task oriented third-person verbal 10k avsd @xcite task oriented third-person verbal 11k igc @xcite task & non-task oriented third-person verbal 4k sdg @xcite non-task oriented previous studies considered non-verbal input information (e.g., human facial expressions), they did not consider the agents' non-verbal responses (i.e., actions). non-verbal responses often play an important role in dialogue systems. for example, a museum tour-guide robot should use nonverbal gestures to explain things to the audience better. even in ordinary conversation, non-verbal responses such as "making a smile" or "helping to lift luggage" are often crucial for social interactions in conjunction with verbal responses. thus, we propose a visually-grounded firstperson dialogue (vfd) dataset with verbal and nonverbal responses. as shown in figure 1 , the vfd dataset comprises (1) first-person images of agents, (2) utterances of human speakers, (3) eye-gaze locations of the speakers, and (4) the agents' verbal and non-verbal responses to the utterances. here, human utterances and agents' verbal and non-verbal responses were manually annotated for first-person images (with eye-gaze locations) in the gazefollow dataset @xcite using crowdsourcing with carefully-designed settings, resulting in 308k verbal and 81k non-verbal dialogues. this paper also presents experimental results obtained using the vfd dataset and recent neural network models, e.g., bert @xcite and resnet @xcite . our primary contributions are summarized as follows. (1) we present a new multimodal dialogue dataset that contains visually-grounded first-person dialogues with human speakers' eye-gaze locations. ## related work several multimodal dialogue datasets have investigated task-oriented situations. for example, mmd dataset @xcite contains dialogues between shoppers of fashion products and sales agents. talkthewalk dataset @xcite aims to guide tourists to their destinations. in visdial dataset @xcite and avsd dataset @xcite , an agent must answer questions about an input image (or video) given dialogue history. unlike these datasets, which can only work in some limited scenarios, we aim to cover both task-oriented and non-task-oriented dialogue systems. as shown in like our vfd dataset, sdg dataset @xcite contains dialogues with non-verbal actions. however, sdg focuses on gestures (or body languages), e.g., "making a cup shape with the right hand", which are categorized into 271 gesture classes. in contrast, vfd dataset represents nonverbal responses as text (typically sentences) to cover a wider range of gestures, e.g., "check out the same clothing size", "buy one of the pumpkins a girl has", etc. in addition, our vfd dataset is large in comparison to other datasets. it is twice the size of the mmd dataset and approximately 75 times the size of the igc dataset. igc dataset is small because it provides only validation and test sets. ## conclusion in this paper, we have presented the vfd dataset with verbal and non-verbal responses. we manually annotated 308k human utterances and 308k verbal and 81k non-verbal responses of agents, which are grounded in the agents' first-person images with human eye-gaze locations. we confirmed the validity of the first-person view in the experiments for the response selection tasks; however, this task (especially, non-verbal response production) remains challenging, and improvements are required. image. the third example from the left in figure 6 (b) is also difficult because the agent has to have the common knowledge that we must put away the used trays before we leave in a cafe. in the rightmost example in figure 6 (b), we consider that the visual information wrongly lets the models take actions related to more specific information about players or shoes rather than the more general action of suggesting to buy the shoes.