Automatically describing a video with natural language is regarded as a fundamental challenge in computer vision. The problem nevertheless is not trivial especially when a video contains multiple events to be worthy of mention, which often happens in real videos. A valid question is how to temporally localize and then describe events, which is known as "dense video captioning." In this paper, we present a novel framework for dense video captioning that unifies the localization of temporal event proposals and sentence generation of each proposal, by jointly training them in an end-to-end manner. To combine these two worlds, we integrate a new design, namely descriptiveness regression, into a single shot detection structure to infer the descriptive complexity of each detected proposal via sentence generation. This in turn adjusts the temporal locations of each event proposal. Our model differs from existing dense video captioning methods since we propose a joint and global optimization of detection and captioning, and the framework uniquely capitalizes on an attribute-augmented video captioning architecture. Extensive experiments are conducted on ActivityNet Captions dataset and our framework shows clear improvements when compared to the state-of-the-art techniques. More remarkably, we obtain a new record: ME-TEOR of 12.96% on ActivityNet Captions official test set. * This work was performed at Microsoft Research Asia. The dog brings the frisbee back to the man. A man and a dog are outdoors and waiting for their turn to play on a fenced in green field. The man and the dog runs onto the field and he throws the frisbee a far distance and the dog runs and fetches it, then returns it back to the man and they repeat the process 6 times. When they are done, another man runs to them and hands the man a leash and he leashes his dog. The whole time there are people on the sidelines watching them and taking pictures. A man and a dog walk onto a field. A man throws a frisbee and the dog chases after it. A man is playing frisbee with a dog.
Li et al. REF propose a descriptiveness regression component to unify the event localization and sentence generation.
Jointly Localizing and Describing Events for Dense Video Captioning
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Computer Science