Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in V ideo QA (2020)

Abstract: Videos convey rich information. Dynamic spatio-temporal relationships between people/objects, and diverse multimodal events are present in a video clip. Hence, it is important to develop automated models that can accurately extract such information from videos. Answering questions on videos is one of the tasks which can evaluate such AI abilities. In this paper, we propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Specifically, we first employ dense image captions to help identify objects and their detailed salient regions and actions, and hence give the model useful extra information (in explicit textual format to allow easier matching) for answering questions. Moreover, our model is also comprised of dual-level attention (word/object and frame level), multi-head self/cross-integration for different sources (video and dense captions), and gates which pass more relevant information to the classifier. Finally, we also cast the frame selection problem as a multi-label classification task and introduce two loss functions, In-andOut Frame Score Margin (IOFSM) and Balanced Binary Cross-Entropy (BBCE), to better supervise the model with human importance annotations. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin (74.09% versus 70.52%). We also present several word, object, and frame level visualization studies.

## introduction recent years have witnessed a paradigm shift in the way we get our information, and a lot of it 1 our code is publicly available at: https://github.com/hyounghk/ videoqadensecapframegate-acl2020 is related to watching and listening to videos that are shared in huge amounts via the internet and new high-speed networks. videos convey a diverse breadth of rich information, such as dynamic spatiotemporal relationships between people/objects, as well as events. hence, it has become important to develop automated models that can accurately extract such precise multimodal information from videos @xcite @xcite @xcite @xcite . video question answering is a representative ai task through which we can evaluate such abilities of an ai agent to understand, retrieve, and return desired information from given video clips. in this paper, we propose a model that effectively integrates multimodal information and locates the relevant frames from diverse, complex video clips such as those from the video+dialogue tvqa dataset @xcite , which contains questions that need both the video and the subtitles to answer. when given a video clip and a natural language question based on the video, naturally, the first step is to compare the question with the content (objects and keywords) of the video frames and subtitles, then combine information from different video frames and subtitles to answer the question. analogous to this process, we apply dual-level attention in which a question and video/subtitle are aligned in word/object level, and then the aligned features from video and subtitle respectively are aligned the second time at the frame-level to integrate information for answering the question. among the aligned frames (which contain aggregated video and subtitle information now), only those which contain relevant information for answering the question are needed. hence, we also apply gating mechanisms to each frame feature to select the most informative frames before feeding them to the classifier. next, in order to make the frame selection more effective, we cast the frame selection sub-task as a multi-label classification task. to convert the time span annotation to the label for each frame, we assign a positive label ('1') to frames between the start and end points, and negative ('0') label to the others, then train them with the binary crossentropy loss. moreover, for enhanced supervision from the human importance annotation, we also introduce a new loss function, in-and-out frame score margin (iofsm), which is the difference in average scores between in-frames (which are inside the time span) and out-frames (which are outside the time span). we empirically show that these two losses are complementary when they are used together. also, we introduce a way of applying binary cross-entropy to the unbalanced dataset. as we see each frame as a training example (positive or negative), we have a more significant number of negative examples than positive ones. to balance the bias, we calculate normalized scores by averaging the loss separately for each label. this modification, which we call balanced binary crossentropy (bbce), helps adjust the imbalance and further improve the performance of our model. finally, we also employ dense captions to help further improve the temporal localization of our video-qa model. captions have proven to be helpful for vision-language tasks @xcite @xcite by providing additional, complementary information to the primary task in descriptive textual format. we employ dense captions as an extra input to our model since dense captions describe the diverse salient regions of an image in object-level detail, and hence they would give more useful clues for question answering than single, non-dense image captions. empirically, our first basic model (with duallevel attention and frame-selection gates) outperforms the state-of-the-art models on tvqa validation dataset (72.53% as compared to 71.13% previous state-of-the-art) and with the additional supervision via the two new loss functions and the employment of dense captions, our model gives further improved results (73.34% and 74.20% respectively). these improvements from each of our model components (i.e., new loss functions, dense captions) are statistically significant. overall, our full model's test-public score substantially outperforms the state-of-the-art score by a large margin of 3.57% (74.09% as compared to 70.52%). 2 also, our model's scores across all the 6 tv shows are more balanced than other models in the tvqa leaderboard 3 , implying that our model should be more consistent and robust over different genres/domains that might have different characteristics from each other. our contributions are four-fold: (1) we present an effective model architecture for the video question answering task using dual-level attention and gates which fuse and select useful spatial-temporal information, (2) we employ dense captions as salient-region information and integrate it into a joint model to enhance the videoqa performance by locating proper information both spatially and temporally in rich textual semi-symbolic format, (3) we cast the frame selection sub-task as a multilevel classification task and introduce two new loss functions (iofsm and bbce) for enhanced supervision from human importance annotations (which could be also useful in other multi-label classification settings), and (4) our model's score on the test-public dataset is 74.09%, which is around 3.6% higher than the state-of-the-art result on the tvqa leaderboard (and our model's scores are more balanced/consistent across the diverse tv show genres). we also present several ablation and visualization analyses of our model components (e.g., the word/object-level and the frame-level attention). ## model our model consists of 2 parts: feature fusion and frame selection. for feature fusion, we introduce dual-level (word/object and frame level) attention, and we design the frame selection problem as a multi-label classification task and introduce 2 new loss functions for enhanced supervision (figure 1 ). ## results and ablation analysis as seen from ## visualizations in this section, we visualize the dual-level attention (word/object and frame level) and the frame score change by new losses application (for all these attention examples, our model predicts the correct answers). ## conclusion we presented our dual-level attention and frameselection gates model and novel losses for more effective frame-selection. furthermore, we employed dense captions to help the model better find clues from salient regions for answering questions. each component added to our base model architecture (proposed loss functions and the adoption of dense captions) significantly improves the model's performance. overall, our model outperforms the state-of-the-art models on the tvqa leaderboard, while showing more balanced scores on the diverse tv show genres.