Hierarchical Context-aware Network for Dense Video Event Captioning (2021)

Abstract: Dense video event captioning aims to generate a sequence of descriptive captions for each event in a long untrimmed video. Video-level context provides important information and facilities the model to generate consistent and less redundant captions between events. In this paper, we introduce a novel Hierarchical Context-aware Network for dense video event captioning (HCN) to capture context from various aspects. In detail, the model leverages local and global context with different mechanisms to jointly learn to generate coherent captions. The local context module performs full interaction between neighbor frames and the global context module selectively attends to previous or future events. According to our extensive experiment on both Youcook2 and Activitynet Captioning datasets, the video-level HCN model outperforms the event-level context-agnostic model by a large margin. The code is available at https://github.com/KirkGuo/HCN .

## introduction with the increase of video data uploaded online every day, the acquisition of knowledge from videos especially for howto tasks is indispensable for people's daily life and work. however, watching a whole long video is time-consuming. existing technologies focus on two main research directions to compact video information: video summarization to trim long videos to short ones and (dense) video captioning to generate a textual description of the key events in the video. typically for long untrimmed videos, dense video event captioning generates fine-grained captions for all events to facilitate users quickly skimming the video content and enables various applications e.g. video chaptering and search inside a video. ## preliminary the dense video event captioning task is to produce a sequence of events and generate a descriptive sentence for each event given a long untrimmed video. in this work, we focus only on the task to generate captions and directly apply the ground-truth event proposals similar to @xcite . the paradigm for video captioning is an encoder-decoder network, which inputs video features and outputs descriptions for each event. in this section, we describe the task formulation including the context-agnostic model as well as the context-aware model in one framework. ## conclusion and discussion dense video event captioning is a typical video understanding task to learn procedural events in a long untrimmed video. it is essential to model holistic video information for event understanding. in this paper, we propose a novel hierarchical context-aware network to encode both the local and global context of long videos. our hcn model is effective in modeling context and outperforms the context-agnostic model by a large margin. in future work, we tend to extend our hierarchical network to further investigate how to effectively attend to the long context to filter ambiguous and irrelevant information.