HEAR : Hearing Enhanced Audio Response for Video-grounded Dialogue (2023)

Abstract: Video-grounded Dialogue (VGD) aims to answer questions regarding a given multi-modal input comprising video, audio, and dialogue history. Although there have been numerous efforts in developing VGD systems to improve the quality of their responses, existing systems are competent only to incorporate the information in the video and text and tend to struggle in extracting the necessary information from the audio when generating appropriate responses to the question. The VGD system seems to be deaf, and thus, we coin this symptom of current systems’ ignoring audio data as a deaf response. To overcome the deaf response problem, Hearing Enhanced Audio Response (HEAR) framework is proposed to perform sensible listening by selectively attending to audio whenever the question requires it. The HEAR framework enhances the accuracy and audibility of VGD systems in a model-agnostic manner. HEAR is validated on VGD datasets (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows effectiveness with various VGD systems.

## introduction one of the desiderata in our vision-language community is to build conversational agents that can look, listen, think and speak as humans. these agents can potentially be deployed in various subsections of society, including education, security, entertainment, and visual or other impairments. to promote natural conversation between humans and the conversational agents, the video-grounded dialogue (vgd) task @xcite has been designed, aiming to respond to the questions regarding a given multimodal input comprising video, audio, and dialogue history. to be specific, as shown in figure 1 , given video v , audio u , dialogue history composed of caption c and past rounds of q&a pairs @xmath0, (q 1 , a 1 ), ..., (q r-1 , a r-1 )}, and current r-th he sets the remote down on the bed. ## hearing enhanced audio response figure 3 illustrates hearing enhanced audio response (hear) framework designed to enhance dialogue language model (dlm) 2 in terms of two functionalities on audio: (1) sensibility that selectively attends on audio according to the meaning of the question and (2) audibility that performs enhanced listening to input audio. for sensibility, we propose sensible audio listening (sal) in figure 3 (a) which trains the dlm to respond to a question by selectively weighting to audio corresponding to the audio-relatedness of the question. for audibility, we devise reconstructive listening enhancement (rle) in figure 3 (b) which enhances audio representations by establishing a reconstruction upper bound to connect audio with its surrounding information. we alternately train dlm with sal and rle to fully utilize video and audio modalities.