Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos (2020)

Abstract: Multimodal summarization for open-domain videos is an emerging task, aiming to generate a summary from multisource information (video, audio, transcript). Despite the success of recent multiencoder-decoder frameworks on this task, existing methods lack fine-grained multimodality interactions of multisource inputs. Besides, unlike other multimodal tasks, this task has longer multimodal sequences with more redundancy and noise. To address these two issues, we propose a multistage fusion network with the fusion forget gate module, which builds upon this approach by modeling fine-grained interactions between the modalities through a multistep fusion schema and controlling the flow of redundant information between multimodal long sequences via a forgetting module. Experimental results on the How2 dataset show that our proposed model achieves a new state-of-the-art performance. Comprehensive analysis empirically verifies the effectiveness of our fusion schema and forgetting module on multiple encoder-decoder architectures. Specially, when using high noise ASR transcripts (WER>30%), our model still achieves performance close to the ground-truth transcript model, which reduces manual annotation cost.

## introduction with the popularity of video platforms, personal videos abound on the internet. multimodal summarization for open-domain videos, first organized as a track of the how2 challenge at the icml 2019 workshop, aims to integrate multisource information of videos (video, audio, transcript) into a fluent textual summary. an example can be seen in figure 1 . this study, which uses compressed text description to reflect the salient parts of videos, * corresponding author. existing approaches have obtained promising results. for example, @xcite and @xcite utilize multiple encoders to encode videos and audio transcripts and a joint decoder to decode the multisource encodings, which acquire better performance than single modality structures. despite the effectiveness of these approaches, they only perform multimodal fusion during the decoding stage to generate a target sequence, lacking fine-grained interactions between multisource inputs to complete the missing information of each modality. for example, as shown in figure 1 , text context representations containing birds should be associated with visual semantic information containing parrots to build thorough multimodal representations. besides, unlike other multimodal tasks such as visual question answering @xcite and multimodal machine translation @xcite , a major challenge is that this task has longer input sequences with more noise and redundancy. the flow of noise information during multimodal fusion, such as redundant frames in video and noisy words in transcription, interferes with the interaction and complementarity of the effective information between modalities, which leads to a significant negative effect on the model. moreover, when using an automatic speech recognition (asr) system to transform audio to transcription instead of ground-truth transcription, high noise asr-output transcripts further reduce model performance. to address these two issues, we propose a multistage fusion network with the fusion forget gate module for multimodal summarization in videos. the model involves multiple information fusion processes to capture the correlation between multisource modalities spontaneously, and a fusion for-in this clip we 're going to file allister 's nails down with a drill . that will help smooth out the nails once again making it comfortable for you to hold your bird as well as comfortable for him . you do n't want any sharp edges on there . it just files it down . you want to use a medium speed on the drill so that you have control over it and it 's not going too fast . and all we 're doing with this is taking off the very tip of the nail after we 've trimmed it just to smooth it out . we 're not going to need to do much with it other than to make it smooth so it 's comfortable to hold him . the bird does not get hurt by the drill but you do want to make sure that the other toes are out of the way of the drill so that the drill piece is not going against their skin . once again , that can be difficult to do , you have to pry their toes apart to get them opened to drill them . ## related work unlike conventional summarization @xcite @xcite , multimodal summarization compresses multimedia documents. according to different tasks, the input modalities are also different, such as text+image @xcite @xcite , and video+audio+text @xcite , which mainly focus on extractive approaches. with the popularity of sequence-to-sequence learning @xcite , the use of corpora with human-written summaries for multimodal abstractive summarization has attracted interest @xcite @xcite . the above abstractive summarization research mainly focuses on text and image. @xcite first release the how2 dataset for multimodal abstractive summarization for open-domain videos. the dataset provides multisource information, including video, audio, text transcription and human-generated summary. this task is more challenging due to the diversity of multimodal information in the video and the complexity of the video feature space. the task was also added to the how2 challenge in the 2019 icml workshop, ## multistage fusion with forget gate in this section, we will explain our model in detail. the overall architecture of our proposed model is shown in figure 2 , and the fusion forget gate inside is illustrated in figure 3 . specifically, multistage fusion consists of the cross fusion block and hierarchical fusion decoder, which aims to model the correlation and complementarity between modalities spontaneously. in addition, the fusion forget gate is applied in the cross fusion block to filter the flow of redundant information streams. we build our model based on the rnn and transformer encoder-decoder architectures, respectively. ## conclusions we introduce a multistage fusion network with fusion forget gate for generating text summaries for the open-domain videos. we propose a multistep fusion schema to model fine-grained interactions between multisource modalities and a fusion forget gate module to handle the flow of multimodal noise of multisource long sequences. experiments on the how2 dataset show the effectiveness of the proposed models. furthermore, when using high noise speech recognition transcription, our model still achieves the effect of being close to the groundtruth transcription model, which reduces the manual annotation cost of transcripts. compute a new hidden state h t , which is defined as: