An Efficient Keyframes Selection Based Framework for Video Captioning (2021)

Abstract: Describing a video is a challenging yet attractive task since it falls into the intersection of computer vision and natural language generation. The attention-based models have reported the best performance. However, all these models follow similar procedures, such as segmenting videos into chunks of frames or sampling frames at equal intervals for visual encoding. The process of segmenting video into chunks or sampling frames at equal intervals causes encoding of redundant visual information and requires additional computational cost since a video consists of a sequence of similar frames and suffers from inescapable noise such as uneven illumination, occlusion and motion effects. In this paper, a boundary-based keyframes selection approach for video description is proposed that allow the system to select a compact subset of keyframes to encode the visual information and generate a description for a video without much degradation. The proposed approach uses 3 4 frames per video and yields competitive performance over two benchmark datasets MSVD and MSR-VTT (in both English and Hindi).

## introduction in recent years, we witnessed the exponential growth in multimedia data (especially video) over the internet @xcite . this large volume of data creates a need for automatic video under standing systems that can describe the video's con tent, event and action with a short textual descrip tion. there are many applications of automatic video description generation such as efficient con tent indexing and searching, storytelling, the amal gamation of speech with the video description can also help visually impaired people and if the video description approaches are successful in generat ing a short textual description of the realworld scenes, then the robots can converse with humans effectively @xcite . the task of generating image and video descrip tions are very closely related. but the presence of both temporal and spatial information, which varies with the time in a video, makes the task of video description generation more challenging than image description. so for generating an infor mative and visually related video description, the efficient encoding of both spatial and temporal fea tures of the video is the basic step in any video de scription framework. being an interdisciplinary problem of both com puter vision and natural language processing, re searchers form both domain have proposed a nu merous approach for describing a video precisely, but still, much work is needed to be done. a video consists of a sequence of similar frames, but var ious editing effects are included in the video due to the recent advancements in technologies and these editing effects affect the process of select ing informative frames from the video. existing approaches such as @xcite encode the visual features of the video either by segmenting the video in the interval of some arbitrary value k (most of time @xmath0. meanwhile, the pro cess of encoding visual features by equal interval sampling does not guarantee that all the selected frames are informative because, in a video it is pos sible that the selected frames are suffering from dif ferent types of noise such as uneven illumination, motion blur, occlusion and object zoomin/out ef fects @xcite . in this paper, we ad dress the issue of selecting informative frames by using color information based shot boundary de tection followed by keyframe selection from each shot. a shot in a video is a set of continuous similar frames captured uninterruptedly and when the content of these frames get changed, it creates two types of boundaries (transitions) in the video namely abrupt and gradual transition. the novel contribution of the proposed work are: the organization of the remaining part of the paper is as follows. section 2 report a review of related work on video description. section 3 discussed the proposed approach. a detail experimental studies is reported in section 4 followed by conclusion in section 5. ## related work earlier, the process of bridging the gap between visual content and natural language was consid ered a challenging task. however, with the suc cess of deep learning approaches in recent years, the gap has been reduced. till now, the approaches proposed for video description can be categorised into three phases: classical method based phase, statistical method based phase and deep learning based phase @xcite . further, the related work in this section is divided into three subsections based on the type of approach is em ployed: sequencetosequence based approaches (s2s), attentionbased approaches and boundary based approaches. ## proposed approach the proposed approach consists of two modules: boundary detection and keyframe selection mod ule (sec 3.1) and description generation module (sec 3.2). ## conclusion in this paper, we employ a boundaryaware keyframe selection framework that acts as a plug andplay module for downstream videorelated tasks, such as video description and video clas sification. the objective of the boundary aware keyframe selection framework is to select a com pact subset of keyframes for input video, which minimises the unnecessary processing of visually similar frames and ensures no degradation in the quality generated description. in the proposed ap proach, 3 ∼ 4 frames are selected for an input video, which is more efficient than the existing p ickn et model, which picks 6 ∼ 8 frames for each video. the experimental results show that the keyframesbased approach can outperform exist ing methods by picking keyframes and extracting different visual features such as appearance, mo tion and region features.