Despite recent progress on computer vision and natural language processing, developing video understanding intelligence is still hard to achieve due to the intrinsic difficulty of story in video . Moreover, there is not a theoretical metric for evaluating the degree of video understanding  . In this paper, we propose a novel video question answering (Video QA) task, DramaQA, for a comprehensive understanding of the video story. The DramaQA focused on two perspectives: 1) hierarchical QAs as an evaluation metric based on the cognitive developmental stages of human intelligence. 2) character-centered video annotations to model local coherence of the story. Our dataset is built upon the TV drama "Another Miss Oh" and it contains 16,191 QA pairs from 23,928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217,308 annotated images with rich character-centered annotations, including visual bounding boxes, behaviors , and emotions of main characters, and coreference resolved scripts. Additionally, we provide analyses of the dataset as well as Dual Matching Multistream model which effectively learns character-centered representations of video to answer questions about the video. We are planning to release our dataset and model publicly for research purposes and expect that our work will provide a new perspective on video story understanding research.
Despite recent progress on computer vision and natural language processing, developing a machine that can understand video story is still hard to achieve due to the intrinsic difficulty of video story . Moreover, researches on how to evaluate the degree of video understanding based on human cognitive process have not progressed as yet . In this paper, we propose a novel video question answering (Video QA) task, DramaQA, for a comprehensive understanding of the video story. The DramaQA focuses on two perspectives: 1) Hierarchical QAs as an evaluation metric based on the cognitive developmental stages of human intelligence. 2) Character-centered video annotations to model local coherence of the story. Our dataset is built upon the TV drama "Another Miss Oh" and it contains 17,983 QA pairs from 23,928 various length video clips, with each QA pair belonging to one of four difficulty levels. We provide 217,308 annotated images with rich character-centered annotations, including visual bounding boxes, behaviors and emotions of main characters, and coreference resolved scripts. Additionally, we suggest Multi-level Context Matching model which hierarchically understands character-centered representations of video to answer questions . We release our dataset and model publicly for research purposes , and we expect our work to provide a new perspective on video story understanding research.
R
video understanding intelligence
a machine that can understand video story
R
story in video
video story
R
there is not a theoretical metric for evaluating
researches on how to evaluate
A
based on human cognitive process have not progressed as yet
R
focused
focuses
R
hierarchical
Hierarchical
R
character-centered
Character-centered
R
16,191
17,983
D
,
R
provide analyses of the dataset as well as Dual Matching Multistream model which effectively learns
suggest Multi-level Context Matching model which hierarchically understands
R
about the video. We are planning to
. We
R
and expect that our work will
, and we expect our work to