Sparse Frame Grouping Network with Action Centered for Untrimmed Video Paragraph Captioning (2023)

Abstract: Generating paragraph captions for untrimmed videos without event annotations is challenging, especially when aiming to enhance precision and minimize repetition at the same time. To address this challenge, we propose a module called Sparse Frame Grouping (SFG). It dynamically groups event information with the help of action information for the entire video and excludes redundant frames within pre-defined clips. To enhance the performance, an Intra Contrastive Learning technique is designed to align the SFG module with the core event content in the paragraph, and an Inter Contrastive Learning technique is employed to learn action-guided context with reduced static noise simultaneously. Extensive experiments are conducted on two benchmark datasets (ActivityNet Captions and YouCook2). Results demonstrate that SFG outperforms the state-of-the-art methods on all metrics.

## introduction video paragraph captioning for untrimmed video aims to describe multiple events in the video with three or four sentences. initially, the focus was on describing each event with a single sentence in a long video (j. lei and mohit, 2020) (j. mun and @xcite , until (y. song and @xcite proposed to directly generate paragraph captions without event ground truths. this refined approach to paragraph captioning is more practical but also more challenging. firstly, without event labels, it becomes difficult to precisely locate the transitions between different events and to summarize the complete story from a holistic perspective. secondly, most prior studies generated one sentence for each event, whereas paragraph captioning without event ground truths requires generating sentences that are two or three times longer. this naturally exacerbates issues of redundancy and repetition. our work focuses on video paragraph captioning without event annotations as shown in fig. 1 . as a common frame sampling strategy in most existing works, random sampling method randomly selects one frame every one or two seconds. the hypothesis of previous works is that the frames in one second are highly similar, so random sampling does not result in significant information loss. however, actions often occur rapidly, such as within a one-second time frame, and usually lead changes to events. random sampling may miss crucial action frames that occur quickly and retain many similar frames as noise. according to (j. @xcite , one or two key frames are sufficient to capture important content, and the "redundant" frames can be neglected in an action clip. in this way, we can learn the visual content from a small number of frames that contain critical information. therefore, our objective is to learn event distributions guided by action frames, and utilize them as estimations for event locations, enabling us to construct comprehensive paragraph stories by incorporating these momentary yet important actions. considering the impracticality of including all frames (often averaging hundreds) into the analysis, it becomes essential to find an effective way for frame selection and information grouping. our work specifically focuses on action-driven video paragraph captioning. it tackles the captioning task by utilizing sparse frames and refined video an-chors with action visual representation. no event annotation or any other additional information is used except for the video and paragraph captions, thereby ensuring the practicality of our method. we propose our vanilla model based on the transformer architect (j. @xcite . a new grouping module, named sparse frame grouping (sfg), is introduced. it uses a set of designed anchors as queries to select and group useful frames. the objective is to extract two types of content from the video: event distributions and action changes between adjacent frames. the idea is to progressively estimate the actual event distributions by grouping visually similar video frames using these designed anchors. at each encoder layer, sfg is first applied to every two frames and selects one informative frame, where anchors are random frames. as a result, a sequence of sparse and important video frames are obtained, instead of using the randomly sampled ones. this process is referred to as clip-level sfg. next, sfg is employed to group event context across the entire video. video anchors are initialized using triplets of ground truths during the training phase. the objective is to ensure that the anchors capture the action-driven event distributions in the video, while the triplets provide a concise representation of the main action content in the sentences. this process is referred to as video-level sfg. at this stage, the video anchors are updated by grouping the newly selected frames. the remaining steps follow the standard transformer computation. in the decoder, we modify the cross-attention mechanism by attending video anchors in the video-level sfg additionally. the weighted anchors, along with the weighted video frames, guide the decoder to generate each word at each decoding step. to train sfg without using event annotations, we introduce an intra-and inter-contrastive learning technique. the objective is to ensure that the video anchors are close to their relevant event contents in the video. this process naturally involves contrastive learning between anchors and paragraph. the challenge is that the embedding of the entire paragraph contains excessive noise for the video anchors which are intended to represent the center of events. what we aim to learn is a summary of the event context within the paragraph, while disregarding those static descriptions that are not relevant to the actions in our video anchors. we modify the positive contrastive pair from visual-paragraph to visual-triplets, while the negative pairs consist of unmatched pairs within the batch. for each sentence in the paragraph, triplet (s, v, o) (sec. 3.2) is extracted and the embeddings of all triplets are averaged. this contrastive learning approach is referred to as intra-contrastive learning, and its purpose is to minimize the distance between the overall video anchors and the paragraphs. besides, it is crucial to differentiate the video anchors from other visual contents in the video. the inter-contrastive learning method takes the left video representation as a negative visual part. it requires the distance between the video anchors and the triplet embeddings is at least smaller than the distance from the negative visual part to the triplet embeddings. we conduct experiments on activitynet caption and youcook2 datasets. the results demonstrate that our model outperforms the state-of-the-art (sota) methods when evaluated using the event ground truths on both datasets. the contributions of this work are as follows: ## related work dense video captioning as a multi-modal task, it has attracted much attention. popular categorization of dense video captioning falls into three: predicting event location and sentences, solely predicting sentences with event annotations, and predicting paragraphs without event annotations. in this paper, our focus falls into the third category. (l. @xcite is the first adopting transformer to decode sentences and create differentiable masks for the consistency between proposals and captions. (j. mun and han, 2019) used a pointer network to select possible event proposals from candidates and to recurrently generate sen-tence for each event. (t. wang and luo, 2021) firstly tackled this task as a parallel multitask. localization and caption generation are predicted at the same time using a detr scheme. (p. jae sung and anna, 2019) adopted adversarial inference to evaluate the paragraph quality. (j. lei and mohit, 2020) proposed a memory module in decoder to remove redundant visual and text information dynamically. these prior studies all rely on event annotations. (y. song and jin, 2021) focused on reducing redundancy using an adding and erasing module at each decoding step without event annotations. however, little attention have been paid to video redundancy itself at the encoder stage. in our work, we aim to address this issue by tackling it from the initial frame selection stage and capturing the core action content of the video. video action understanding action localization in video is an important task for video understanding. @xcite and @xcite utilized reinforce learning to dynamically determine the next frame. (z. yuan and gangshan, 2021) proposed to sample motion from cumulative distribution. (b. @xcite proposed to understand video only through some salient clips which were ranked and selected for further understanding. @xcite selected single frame and utilized high level feature for video classification. the previous works seldom combine the two modalities (visual and text) together. our work utilizes both to better select key frames. ## methods the proposed sparse frame grouping network is shown in fig. 2 . it consists of three modules: cliplevel sfg, video-level sfg, and intra-and inter-contrastive learning. together with the vanilla transformer, these modules generate action-driven and event annotation-free paragraphs. in this section, the vanilla transformer is first described, followed by the introduction to the three modules. ## qualitative results fig. 4 displays example captions produced by our model on activitynet captions. we compare them with gt (the ground truth) and those generated by the vanilla models. the triplets in all paragraphs are highlighted for comparison. compared to vanilla, our model tends to generate more precise and action-driven descriptions, as evident in the examples. particularly in the third case, where the surroundings are relatively static and only the athlete is performing different actions, our model is able to differentiate between these actions in the neighbouring frames. for event distributions, in the first case, there are three events, and the second is momentary compared with the other two. the vanilla model fails to describe the second event even with annotation information. our model, however, captures the action "is taken by" and decodes it to "give" in the generated description. ## conclusions in this work, we present a network with sparse frame grouping (sfg) enhanced transformer via contrastive learning for event annotation-free video paragraph captioning. the sfg module is designed to select and group informative frames and learn event distributions without relying on event annotations. additionally, an intra-and inter-contrastive learning technique is proposed to train the sfg, enabling it to learn the overall event context in the video and distinguish it from the static context. experimental results on two benchmark datasets demonstrate that sfgtransformer achieves sota performance. in the future, we plan to extend the method to longer video data such as movie data to further investigate the influence of key action visual information. and we also investigate training method without triplets to simplify the pre-process of data.