W hy A ct: Identifying Action Reasons in Lifestyle Vlogs (2021)

Abstract: We aim to automatically identify human action reasons in online videos. We focus on the widespread genre of lifestyle vlogs, in which people perform actions while verbally describing them. We introduce and make publicly available the WhyAct dataset, consisting of 1,077 visual actions manually annotated with their reasons. We describe a multimodal model that leverages visual and textual information to automatically infer the reasons corresponding to an action presented in the video.

## introduction significant research effort has been recently devoted to the task of action recognition @xcite @xcite @xcite . action recognition works well when applied to well defined/constrained scenarios, such as people following scripts and instructions @xcite @xcite , performing sports @xcite or cooking @xcite @xcite . at the same time however, action recognition is limited and error-prone once the application space is opened to everyday life. this indicates that current action recognition systems rely mostly on pattern memorization and do not effectively understand the action, which makes them fragile and unable to adapt to new settings @xcite . research on how to improve action recognition in videos @xcite shows that recognition systems for actions with known intent have a significant increase in performance, as knowing the reason for performing an action is an important step for understanding that action @xcite . in contrast to action recognition, action causal reasoning research is just emerging in computational applications @xcite @xcite . causal reasoning has direct applications on many reallife settings, for instance to understand the consequences of events (e.g., if "there is clutter," "cleaning" is required), or to enable social reasoning (e.g., when "guests are expected," "cleaning" may be needed -see figure 1 ). most of the work to date on causal systems has relied on the use of semantic parsers to identify reasons @xcite , however this approach does not work well on more realistic every-day settings. as an example, consider the statement "this is a mess and my friends are coming over. i need to start cleaning." current causal systems are unable to identify "this is a mess" and "friends are coming over" as reasons, and are thus failing to use them as context for understanding the action of "cleaning." in this paper, we propose the task of multimodal action reason identification in everyday life scenarios. we collect a dataset of lifestyle vlogs from youtube that reflect daily scenarios and are currently very challenging for systems to solve. vloggers freely express themselves while performing most common everyday activities such as cleaning, eating, cooking, writing and others. lifestyle vlogs present a person's everyday routine: the vlogger visually records the activities they perform during a normal day and verbally express their intentions and feelings about those activities. because of these characteristics, lifestyle vlogs are a rich data source for an in depth study of human actions and the reasons behind them. the paper makes four main contributions. first, we formalize the new task of multimodal action reason identification in online vlogs. second, we introduce a new dataset, whyact, consisting of 1,077 (action, context, reasons) tuples manually labeled in online vlogs, covering 24 actions and their reasons drawn from conceptnet as well as crowdsourcing contributions. third, we propose several models to solve the task of human action reason "just put everything in everybody is drawers and slowly but surely we are down to the bedding oh i always do it on a strip to bed because i usually clean the bedding the same days i do the laundry .." ## related work there are three areas of research related to our work: identifying action motivation, commonsense knowledge acquisition, and web supervision. ## data collection and annotation in order to develop and test models for recognizing reasons for human actions in videos, we need a manually annotated dataset. this section describes the whyact dataset of action reasons. ## identifying causal relations in vlogs given a video, an action, and a list of candidate action reasons, our goal is to determine the reasons mentioned or shown in the video. we develop a multimodal model that leverages both visual and textual information, and we compare its performance with several single-modality baselines. the models we develop are unsupervised in that we are not learning any task-specific information from a training dataset. we use a validation set only to tune the hyper-parameters of the models. ## evaluation we consider as gold standard the labels selected by the majority of workers (at least two out of three workers). for our experiments, we split the data across video-clips: 20% development and 80% test (see experiments on whyact reveal that both textual and visual modalities contribute to solving the task. the results demonstrate that the task is challenging and there is room for improvement for future work models. selecting the most frequent reason for each action on test data achieves on average an f1 of 40.64, with a wide variation ranging from a very low f1 for the action "writing" (7.66 f1) to a high f1 for the action "cleaning" (55.42 f1). note however that the "most frequent reason" model makes use of data distributions that our models do not use (because our models are not trained). furthermore, we believe that it is expected that for certain actions the distribution of reasons is unbalanced, as in everyday life there are action reasons much more common than others (e.g. for "cleaning", "remove dirt" is a more common/frequent reason than "company was coming"). ## data analysis we perform an analysis of the actions, reasons and video-clips in the whyact dataset. the distribution of actions and their reasons are shown in figure 4 . the supplemental material includes additional analyses: the distribution of actions and their number of reasons (figure 11 ) and videos (figure 10 ) and the distribution of actions and their worker agreement scores (figure 9 ). we also explore the content of the videos by analysing their transcripts. in particular, we look at the actions and their direct objects. for example, the action clean is depicted in various ways in the videos: "clean shower", "clean body", "clean makeup", "clean dishes". the action diversity assures that the task is challenging and complex, trying to cover the full spectrum of everyday activities. in figure 5 we show what kind of actions are depicted in the videos: we extract all the verbs and their most five most frequent direct objects using spacy @xcite and then we cluster them by verb and plot them using t-distributed stochastic neighbor embedding (t-sne) (van der @xcite . finally, we analyse what kind of information is required for detecting the action reasons: what is verbally described, visually shown in the video or the combination of visual and verbal cues. for this, we analyse the worker's justifications for selecting the action reasons: if the reasons were verbally mentioned in the video, visually shown or both. for each video, we take the justification selected by the majority of workers. we find that the reasons for the actions can be inferred only by relying on the narration for less than half of the videos (496 / 1,077). for the remaining videos, the annotators answered that they relied on either the visual information (in 55 videos) or on both visual and audio information (in 423 videos). the remaining 103 videos do not have a clear agreement among annotators on the modality used to indicate the action reasons. we believe that this imbalanced split might be a reason for why the multimodal model does not perform as well as the text model. for future work, we want to collect more visual data that contains action reasons. ## conclusion in this paper, we addressed the task of detecting human action reasons in online videos. we explored the genre of lifestyle vlogs, and constructed why-act -a new dataset of 1,077 video-clips, actions and their reasons. we described and evaluated several textual and visual baselines and introduced a multimodal model that leverages both visual and textual information. we built whyact and action reason detection models to address two problems important for the advance of action recognition systems: adaptability to changing visual and textual context, and processing the richness of unscripted natural language. in future work, we plan to experiment with our action reason detection models in action recognition systems to improve their performance. the dataset and the code introduced in this paper are publicly available at https://github.com/ michigannlp/vlog_action_reason .