We tackle the problem of video object codetection by leveraging weak semantic constraints implied by sentences that describe the video content. Unlike most existing work that focuses on codetecting large objects which are usually salient both in size and appearance, we can codetect objects that are small or medium. Also our method assumes no human pose or depth information which is exploited by the most recent state-of-the-art method. We employ weak semantic constraints on the codetection process by pairing the video with sentences. Although the semantic information is usually simple and weak, it can greatly boost the performance of our codetection framework by reducing the search space of the hypothesized object detections. Our experiment demonstrates an average IoU score of 0.43 on a new challenging dataset which contains 15 distinct object classes and 150 videos with 12,509 frames in total.
REF grounded objects in constrained videos by leveraging weak semantic constraints implied by a sequence of sentences.
Sentence Directed Video Object Codetection
ArXiv
ArXiv
Computer Science