V i LPA ct: A Benchmark for Compositional Generalization on Multimodal Human Activities (2023)

Abstract: We introduce \texttt{ViLPAct} , a novel vision-language benchmark for human activity planning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their initial activities and intents in text. The dataset consists of 2.9k videos from \texttt{Charades} extended with intents via crowdsourcing, a multi-choice question test set, and four strong baselines. One of the baselines implements a neurosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (SOTA) methods. According to our extensive experiments, the key challenges are compositional generalization and effective use of information from both modalities.

## dataset construction we adopt videos from charades @xcite and solicit intents for videos via crowdsourcing. we consider videos that have action sequences of sufficient length appearing in both initial video clips and answers, which result in a dataset comprising 2,912 videos. the dataset is split into training/validation/test sets with a ratio of 70%, 10%, 20%. on the training dataset, we build an mkb by incorporating structural and conceptual information. on the test dataset, we collect a set of mqas for model evaluation. the evaluation with mqas is in fact an adversarial testing method, widely used for quality estimation in machine translation @xcite . herein, the ability of a model to discriminate between correct outputs and meaning-changing perturbations is predictive of its overall performance, not just its robustness. thus mqas are applied only for testing. ## baselines vl plannning of human activities requires predicting future action sequences given an initial visual state video and an intent provided in textual form. the task poses two major challenges. first, information provided in two modalities are complementary to each other, while the majority of multimodal research focuses on the shared information by exploring fusion techniques @xcite . second, the output space is exponentially large with respect to the action space. it is not realistic to assume that all action sequences are already observed in the training data. hence, any models to tackle this task are expected to address systematic composition @xcite of human activities, the capacity to understand and produce a huge number of novel combinations of known actions. in contrast, state-of-the-art deep learning methods often perform poorly on compositional generalization (lake, 2019; @xcite . we compare deep generative models and a neurosymbolic planning model in the framework of retrieval and reasoning. given the first 20% of a video and a future-directed intent, the first step is to obtain top-k relevant action sequences, followed by performing reasoning over the top-k action sequences to find the most plausible answers. both types of models share the same reasoning module but differ in how they obtain top-k action sequences. for reproducibility, the details of all models are provided in appendix 7.3 and 7.4. ## experiments we conduct extensive experiments to answer the following three main research questions. the other research questions are addressed in appendix 7.9. rq1: how reliable is the mqa evaluation method? we show that the evaluation results using mqa are consistent with those by asking humans to directly observe model outputs. for this, we recruit five crowd-workers to rank all models in comparison on each of the 100 questions randomly sampled from the test set, and compare them with the corresponding results using mqa. specifically, for each question, a crowd-worker is asked to rank the top-k outputs of the four baselines in terms of how well they match the intent and the remaining 80% of the original videos. as a result, figure 4 shows how frequent each model is ranked at position x judged by the crowd-workers w.r.t. the top-10 predictions (left) and top-1 predictions (right), respectively. in both cases, we consistently find that the best model is act-univl, followed by act-prophetnet, nsplan, and twostageplan. the ranking result is the same as using mqa on the same set of questions. the ranking differences on individual questions between the human evaluation and mqa are statistically insignificant according to wilcoxon's signed-rank test @xcite , details of which can be found in appendix 7.8. rq2: what are the key challenges? we identify two major challenges of the target task. compositional generalization using reasoning. it is common practice to rank each answer by the likelihood yielded by a generative model @xcite . however, effective use of both modalities. to understand the utility of each modality, we compare the two strongest multimodal models by varying their inputs: including both modalities or just a single modality. as shown in the visual features of act-univl for randomly selected ones during both training and inference, finding that act-univl suffers from only a 4% drop of accuracy using mqa. hence, the multimodal models capture only weak associations between visual features and future action sequences. it is counter-intuitive that visual features do not play a significant role, because plans vary in accordance with different visual environments. we conjecture this is due to poor performance of action recognition. to verify this, we feed ground-truth actions observed in the first 20% of videos to both twostageplan and nsplan during training and inference. they reach an accuracy of 82.11% and 81.37% respectively, improved by more than 15%. rq3: to what degree can the top-k results reflect the performance differences of systems? the reasoning method probinf leverages the top-k results produced by the models, hence it is useful to inspect those results for further insights. therefore, we compare the top 10 results of each model in terms of precision and recall by treating each action sequence as a set (ng and fernando, 2020), as well as seq-hits@5 for measuring exactly matched action sequences. moreover, to investigate the diversity of the top-k lists, we consider dist1 and dist2 @xcite , which respectively measure the number of unique action and consecutive action pairs in the top-k lists. the definitions of a complete list of used metrics and their results are provided in appendix 7.6 and 7.4.1. according to ## conclusion we construct the novel benchmark vilpact to evaluate the ability of systems to anticipate and plan human actions in a multimodal visionlanguage setting, with a focus on evaluating their compositional generalization capabilities. in this benchmark, we extend charades with intents, construct a test set with multi-choice questions, and include four strong baselines. our empirical studies demonstrate that the task is easy for humans, but challenging for sota deep learning models due to the need for compositional generalization and an effective use of information from both modalities. the neurosymbolic planning baseline shows a promising research avenue for using symbolic and multimodal knowledge in an mkb.