  Video Question Answering (VidQA) evaluation metrics have been limited to a
single-word answer or selecting a phrase from a fixed set of phrases. These
metrics limit the VidQA models' application scenario. In this work, we leverage
semantic roles derived from video descriptions to mask out certain phrases, to
introduce VidQAP which poses VidQA as a fill-in-the-phrase task. To enable
evaluation of answer phrases, we compute the relative improvement of the
predicted answer compared to an empty string. To reduce the influence of
language bias in VidQA datasets, we retrieve a video having a different answer
for the same question. To facilitate research, we construct ActivityNet-SRL-QA
and Charades-SRL-QA and benchmark them by extending three vision-language
models. We further perform extensive analysis and ablative studies to guide
future work.
