With increasing information from social media, there are more and more videos available. Therefore, the ability to reason on a video is important and deserves to be discussed. The Dialog System Technology Challenge (DSTC7) (Yoshino et al. 2018) proposed an Audio Visual Scene-aware Dialog (AVSD) task, which contains five modalities including video, dialogue history, summary, and caption, as a sceneaware environment. In this paper, we propose the entropyenhanced dynamic memory network (DMN) to effectively model video modality. The attention-based GRU in the proposed model can improve the model's ability to comprehend and memorize sequential information. The entropy mechanism can control the attention distribution higher, so each tobe-answered question can focus more specifically on a small set of video segments. After the entropy-enhanced DMN secures the video context, we apply an attention model that incorporates summary and caption to generate an accurate answer given the question about the video. In the official evaluation, our system can achieve improved performance against the released baseline model for both subjective and objective evaluation metrics.
REF adds entropy to the loss function to make the sparse distribution more specifically concentrate on a small set of video segments in the VQA task.
Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
ArXiv
ArXiv
Computer Science