V ideo2 C ommonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020)

Abstract: Captioning is a crucial and challenging task for video understanding. In videos that involve active agents such as humans, the agent’s actions can bring about myriad changes in the scene. Observable changes such as movements, manipulations, and transformations of the objects in the scene, are reflected in conventional video captioning. Unlike images, actions in videos are also inherently linked to social aspects such as intentions (why the action is taking place), effects (what changes due to the action), and attributes that describe the agent. Thus for video understanding, such as when captioning videos or when answering questions about videos, one must have an understanding of these commonsense aspects. We present the first work on generating commonsense captions directly from videos, to describe latent aspects such as intentions, effects, and attributes. We present a new dataset “Video-to-Commonsense (V2C)” that contains ~9k videos of human agents performing various actions, annotated with 3 types of commonsense descriptions. Additionally we explore the use of open-ended video-based commonsense question answering (V2C-QA) as a way to enrich our captions. Both the generation task and the QA task can be used to enrich video captions.

## introduction when humans watch videos they can typically understand and reason about various aspects of the scene beyond the visible objects and actions. this involves understanding that some objects are active agents that not only perform actions and manipulate objects, but are motivated by intentions, have pre-conditions, and that their actions have an effect on the world and their own mental states. for instance, in analyzing the video clip in figure 1 , * equal contribution humans employ various capabilities such as perception, reasoning, inference, and speculation, to come up with a description for the observable sequence of events, but also reason about latent aspects such as the intention of the group of runners "to win the medal", the effect of being "congratulated at the finish line", and the attribute "athletic". the above example also illustrates that recognition of objects, actions, and events is often not enough; understanding causal relationships, social interactions, and commonsense aspects behind them provides context and a more semantic interpretation of the video @xcite . a model that can provide such detailed interpretations facilitates answering inferential questions, such as "will the player get angry later?". however, existing visual understanding systems are unable to perform such tasks that require speculative reasoning. a critical missing element in complex video understanding is the capability of performing commonsense inference, especially a generative model. existing efforts seek to find textual explanations or intentions of human activities as a classification task @xcite or a vision-to-text alignment problem @xcite . in this paper we propose the video to commonsense (v2c) framework to generate visually grounded commonsense descriptions about the underlying event in the video, enriching the factual description provided by a caption. under this framework a system is expected to generate captions as well as three types of commonsense @xcite directly from an input video. the v2c model can also be used as a building block for downstream tasks such as video question answering for questions requiring commonsense. inspired by @xcite , our model -the "v2c-transformer" utilizes: (1) a video encoder to extract global representations of the video, (2) a transformer decoder that generates group of runners get prepared to run a race. ## the v2c dataset for the v2c task we need video clips annotated with commonsense descriptions about the agents in the video, as shown in figure 1 . while there are video captioning datasets such as msr-vtt @xcite , the captions in these datasets describe only the observable objects in the image, but do not describe latent and commonsense aspects. we are the first to curate such a dataset with annotations describing the intention of agent to perform an action, the effect of the action and the attribute of the agent given the action. msr-vtt contains around 10k videos each 10 to 30 seconds long, belonging to 20 categories covering a variety of topics such as sports, music, news, and home videos. each video is accompanied by 20 human-annotated textual descriptions on average. for training and benchmarking the novel v2c task, we further complement msr-vtt with event-level commonsense annotations, i.e. event descriptions with intentions, effects and attributes. we remove captions and videos that do not have clear human activities. this is because having such videos leads to an imbalance in the number of captions for each video, thus making it inappropriate to just evaluate caption generation using bleu scores. atomic @xcite is an atlas of everyday commonsense knowledge and contains 880k triplets about causes and effects of human activities, organized as if-then relations, annotated by crowd-sourced workers. this data can be categorized based on causal relations, thereby giving us the categories "cause", "effect" and "attribute", e.g., "if x wants to relax, then he will play video game." ## experiments in this section we describe the loss function used for training our model, additional details about video pre-processing, hyper-parameters, and baseline models, and the metrics used for evaluation. loss function: the decoder parameters θ are trained to maximize the log-likelihood over the training set given by @xmath0, where setting: in order to obtain video representations, we uniformly sample 40 frames from each video and extract features using feed resnet @xcite pre-trained on imagenet ilsvrc12 dataset @xcite and get a 2048-d output from the last layer. we use one-hot input (1-of-n encoding) of the text input and pass it through an embedding layer to produce a 1028-d hidden vector. we use independent vocabularies for captioning and commonsense generation with sizes 27,603 and 24,010 respectively. note that, as the generated hyperparameters: our decoder is a lightweight transformer decoder consisting of 6 transformer blocks with 8 attention heads each. we use adam optimizer with 5000 warm-up steps, and learning rate initialized at 1e-4, and a dropout probability of 0.1 after the residual layer. our model is trained on a machine with single nvidia 1080-ti gpu. baseline model: we compare our method with strong video captioning baseline models like, s2vt @xcite , "attention-enc-dec" @xcite -lstm based models which reach competitive performing on msr-vtt dataset. and "dense captioning" @xcite , which is a transformer based video captioning model. as "dense captioning" is proposed to generate multiple continuous captions for a long untrimmed videos, we modify this by removing the temporal bounding boxes prediction module, and produce two continuous captions (caption + commonsense sentence) together without corresponded starting and ending time. all baselines are trained to predict commonsense descriptions from video on the v2c dataset. we do not compare with videobert @xcite which is trained on a limited set of cooking videos and hence nontransferable, and requires individual captions for multiple segments of the video. metrics: we report both the performances evaluated by automatic scores and human evaluations following the protocols from @xcite . we evaluate our method using bleu (@xmath1, meteor @xcite , rouge @xcite , and perplexity score of the generation on its corpus. we further conduct human evaluations using amt workers, who are asked to identity whether the generated commonsense justifiably completes the events (v2c-completion). we follow the setup in @xcite and randomly sample 100 videos from test set and collect 10 generations for each. to guarantee the objectiveness of the human evaluations, we hire 5 workers for each sample, yielding 30k ratings in total for each model. ## v2c-qa another way of generating commonsense descriptions about the video is by asking pointed questions. consider the example in 1 where we ask the question "what happens next to the runners", about the effect of the action "prepare" performed by the agents "group of runners" observed in the video. we propose a v2c-qa -an open-ended commonsense video question-answering task, where we ask questions about the intents, effects and attributes of the agents in the video. dataset: we use the caption and commonsense annotations in the v2c dataset to create questionanswer pairs for each video. we first extract the action and subject from the caption using spacy linguistic features @xcite . for each intention, attribute and effect for a video, we use template-based generation to get 7 types ## outlook a video typically contains one or many objects (sometimes performing actions) in different back-grounds, scenes, or situations. some objects may be "passive" such as trees or buildings, while some objects may be "active" such as people performing actions like walking, singing, and driving. this paper is focused on describing such active agents in terms of their intentions, effects of their actions, and attributes that characterize these agents. we distinguish v2c from the traditional video captioning task. video captions describe observable objects, background, and actions, while commonsense descriptions in our task seek to describe the unobservable intentions of the agent (pre-conditions or mental conditions), effects of the action (that happen in the future), and attributes which characterize the agent. thus commonsense generation goes beyond the visible. ours is the first attempt at developing a generative video-based commonsense model. we anticipate that our framework can be utilized for many applications in video understanding, comprehension, human-robot interaction, and learning commonsense in a multimodal setting. ## conclusion in this paper, we explore a novel and challenging task to generate video descriptions with rich commonsense descriptions that complement the factual captions. we expand an existing video captioning dataset for the v2c task through automated retrieval from a textual commonsense corpus followed by human labeling, and present a novel v2c-transformer model to serve as a strong baseline method for the v2c task. our evaluation verifies the effectiveness of our method, while also indicating a scope for further study, enhancement, and extensions in the future. our experiments on using the v2c-transformer as a component for the v2c-qa task show that the model has transfer learning capabilities that can be applied to other vision-andlanguage tasks such as question-answering, that require commonsense reasoning. our dataset creation methodology is a three-step procedure as shown in figure 9 . in the first step, we use the caption to query atomic @xcite and retrieve the top-3 intentions, effects, and attributes. these are re-ranked by a bert based model in the second step. the final step involves humans in the annotation process. we ask human annotators to select the most relevant descriptions, and to provide additional descriptions in their own words. the annotators also convert a subset of our dataset into complete sentence descriptions.