Video Question Answering with Phrases via Semantic Roles (2021)

Abstract: Video Question Answering (VidQA) evaluation metrics have been limited to a single-word answer or selecting a phrase from a fixed set of phrases. These metrics limit the VidQA models’ application scenario. In this work, we leverage semantic roles derived from video descriptions to mask out certain phrases, to introduce VidQAP which poses VidQA as a fill-in-the-phrase task. To enable evaluation of answer phrases, we compute the relative improvement of the predicted answer compared to an empty string. To reduce the influence of language bias in VidQA datasets, we retrieve a video having a different answer for the same question. To facilitate research, we construct ActivityNet-SRL-QA and Charades-SRL-QA and benchmark them by extending three vision-language models. We perform extensive analysis and ablative studies to guide future work. Code and data are public.

## introduction given a video, video question answering (vidqa) requires a model to provide an answer to a video related question. however, existing works treat vidqa as an n-way (n ∼1k) classification task across a fixed set of phrases. models trained under such formulations are strictly restricted in their recall rate, generalize poorly, and have severe limitations for end-user applications. in this work, we introduce video question answering with phrases (vidqap) which treats vidqa as a fill-in-the-phrase task. instead of a question, the input to vidqap consists of a query expression with a query-token. then, given a video, vidqap requires replacing query-token with a sequence of generated words. to generate a query, we leverage video descriptions and assign semantic roles to each phrase in these descriptions. replacing a particular semantic-role with a query token produces a query-answer pair. we illustrate this in figure 1 (details in section 3.1). while free-form answer generation is highly desirable, evaluating them is non-trivial due to two main challenges. first, existing language generation metrics like bleu @xcite or bertscore @xcite operate on sentences rather than phrases. when applied to short phrases, in the absence of context, even close matches like "a person" and "the man" would be falsely rejected due to no n-gram overlap or poor contextual embeddings. second, natural language questions often have strong language priors making it difficult to ascertain if the model retrieved information from the video. to propose a reasonable evaluation metric, we revisit our fill-in-the-phrase formulation. since we know where exactly the generated answer fits in the original query, we can create a complete sentence. with this key insight, we propose relative scoring: using the description as reference sentence, we compute the metrics once by replacing the querytoken once with the predicted answer phrase and once with an empty-string. the model's performance is measured by the relative improvement from the predicted answer compared to the empty string. in particular, substituting the answer phrase in the query expression allows the computing the contextual embeddings required by bertscore. to mitigate the language-bias issue, we emulate the procedure proposed by @xcite where for a given question, another image (or video in our case) is retrieved which has a different answer for the same question. to retrieve such a video, we use a contrastive sampling method @xcite over the dataset by comparing only the lemmatized nouns and verbs within the semantic roles (srls). we then propose contrastive scoring to combine the scores of the two answer phrases obtained from the contrastive samples (details on evaluation in section 3.2). to investigate vidqap, we extend three visionlanguage models namely, bottom-up-top-down @xcite , vognet @xcite and a multi-modal transformer by replacing their classification heads with a transformer @xcite based language decoder. to facilitate research on vidqap we construct two datasets activitynet-srl-qa (asrl-qa) and charades-srl-qa and provide a thorough analysis of extended models to serve as a benchmark for future research (details on model framework in section 3.3 and dataset creation in section 4.1). our experiments validate the merits of mov-ing away from n-way classification, and further show even among sequence generation models there exists a large disparity in performance across semantic-roles (i.e. queries for some roles can be answered very easily compared to other roles). moreover, certain roles hardly benefit from visionlanguage models suggesting room for improvement. finally, we investigate the effects of relative scoring and contrastive scoring for vidqap with respect to bertscore. our contributions in this work are two-fold: (i) we introduce vidqap and propose a systematic evaluation protocol to leverage state-of-art language generation metrics and reduce language bias (ii) we provide extensive analysis and contribute a benchmark on two datasets evaluated using three vision-language models. our code and dataset are publicly available. foot_0 2 related works question answering in images has received extensive attention in part due to its end-user applicability. key to its success has been the availability of large-scale curated datasets like vqa v2.0 @xcite for visual question answering and gqa (hudson and manning, 2019) for relational reasoning. to address the strong language priors, the datasets are balanced by retrieving images which given the same question lead to a different answer. however, these procedures cannot be extended for vidqa since crowd-sourcing to retrieve videos is expensive and there exists no scene-graph annotations for videos. in this work, we perform the retrieval using lemmatized nouns and verbs of the semantic roles labels obtained from video descriptions to balance the dataset. question answering in videos: has garnered less attention compared to imageqa. a major bottleneck is that there is no principled approach to curating a vidqa dataset which reflects the diversity observed in imageqa datasets. for instance, naively crowd-sourcing video datasets leads to questions about color, number which is same as imageqa datasets and doesn't reflect any spatialtemporal structure. to address this issue, tgif-qa @xcite and activitynet-qa @xcite use a question-template to enforce questions requiring spatio-temporal reasoning but forgo the question diversity. an orthogonal approach is to combine vidqa with movie scripts @xcite or subtitles @xcite . however, this severely restricts the domain of videos. moreover, recent works have noted that language-only baselines often outperform vision-language baselines @xcite @xcite . a separate line of related research has focused on scene-aware dialogue @xcite . instead of a single annotator providing both questions and answers, the annotation procedure follows a two-player game setup with one player asking a question and the other player answering with the roles switching after each turn. however, the evaluation method utilizes recall metrics which require the set of phrases to be known apriori. as a result, it doesn't strictly measure the performance of free-form generation but rather how well the ground-truth answer is ranked given a competing set of phrases which is analogous to multiple-choice questions. automatic question generation: due to the above limitations, the dominant approach to create large-scale vidqa dataset has been automatic question generation from existing video descriptions which can be easily crowd-sourced. our proposed formulation of using srls to generate queryexpressions falls in this category. prior works include videoqa @xcite , msr-vtt-qa and msvd-qa @xcite which use a rule based question generator @xcite to convert descriptions to questions and movie-fill-in-the-blanks @xcite which mask outs at most one word which could be a noun, adjective or verb in a sentence. in comparison, our method poses vidqap as fill-in-blanks but with phrases, explicitly asks questions about actions, and the answer phrases are not constrained to a fixed set. as a result of this increased space of phrases, methods on existing datasets cannot be directly applied to vidqap. to enable further research, we contribute two datasets asrl-qa and charades-srl-qa. in srl in vision: has been explored in the context of human object interaction @xcite , situation recognition @xcite , and multi-media extraction @xcite . most related to ours is the usage of srls for grounding @xcite in images and videos @xcite . our work builds on @xcite in using srls on video descriptions, however, our focus is not on grounding. instead, we use srls primarily as a query generation tool and use the argument as a question directive. ## design considerations for vidqap the vidqap task is conceptually simple: given a video and a query expression with a query-token, a model should output an answer phrase that best replaces the query-token. this leads to three main design considerations: (i) how to generate a queryexpression from existing resources (section 3.1) (ii) how to evaluate the answer phrases returned by a model (section 3.2) (iii) what modeling framework choices enable vidqap (section 3.3). ## experiments we briefly discuss the dataset creation process (section 4.1), followed by experimental setup (section 4.2). we then summarize our results (section 4.3) and discuss key-findings. we provide implementation details, qualitative visualizations of our dataset, metrics and trained models in the appendix. ## conclusion in this work, we introduce video question answering with phrases (vidqap) where we pose vidqa as a fill-in-the-phrase task. given a video and query expression, a model needs to compose a sequence of words to answer. we then propose a method to leverage semantic roles from video descriptions to generate query expressions and outline a robust evaluation protocol. this involves computing the relative improvement of the prediction answer compared to an empty string followed by a contrastive sampling stage which reduces language-only biases. we then contribute two datasets asrl-qa and charades-srl-qa to facilitate further on vidqap and benchmark them with three visionlanguage models extended for our proposed task. to assign srls to language descriptions we use allennlp library @xcite which provides an implementation of a bert @xcite based semantic-role labeler @xcite . the system achieves 86.49 f1 score on ontonotes @xcite 5.0 dataset.