Disentangled Action Recognition with Knowledge Bases (2022)

Abstract: Action in video usually involves the interaction of human with objects. Action labels are typically composed of various combinations of verbs and nouns, but we may not have training data for all possible combinations. In this paper, we aim to improve the generalization ability of the compositional action recognition model to novel verbs or novel nouns that are unseen during training time, by leveraging the power of knowledge graphs. Previous work utilizes verb-noun compositional action nodes in the knowledge graph, making it inefficient to scale since the number of compositional action nodes grows quadratically with respect to the number of verbs and nouns. To address this issue, we propose our approach: Disentangled Action Recognition with Knowledge-bases (DARK), which leverages the inherent compositionality of actions. DARK trains a factorized model by first extracting disentangled feature representations for verbs and nouns, and then predicting classification weights using relations in external knowledge graphs. The type constraint between verb and noun is extracted from external knowledge bases and finally applied when composing actions. DARK has better scalability in the number of objects and verbs, and achieves state-of-the-art performance on the Charades dataset. We further propose a new benchmark split based on the Epic-kitchen dataset which is an order of magnitude bigger in the numbers of classes and samples, and benchmark various models on this benchmark.

## introduction understanding human-object interaction is crucial for modeling human behavior, and plays a key role in developing robotic agents that interact with humans. in videos, many of these interactions can be described using the combination of verbs and nouns, e.g, move chair, peel apple. recently researchers have been focusing on the task of compositional action recognition with the goal of recognizing actions represented by such verb-noun pairs. the key challenge comes from the extremely large label space of various combinations. the number of possible verb-noun pairs grows quadratically with respect to the number of verbs and nouns. it is infeasible to collect training data for all possible actions. this motivates us to study the problem of zero-shot compositional action recognition, which aims to predict action with components beyond the vocabularies in train data. to conduct zero-shot learning, we propose our disentangled action recognition with knowledgebases (dark), which leverages knowledge graphs. a knowledge graph (kg) encodes semantic relationships between verb or noun nodes. we apply graph convolutional network (gcn) on kg to predict classifier weights for unseen nodes in graphs. previous work @xcite has explored using knowledge graphs for zero-shot compositional action learning. however, their model builds a graph containing verb, noun and compositional action nodes (which link verbs and nouns). it learns features of novel action nodes by propagating information from connected verb/noun nodes. the number of compositional action nodes during training is in the order of o(n 2 ), and the memory consumption may become prohibitively expensive as we scale this approach to large vocabularies. to overcome this issue, we propose to learn separate classifiers for verbs and nouns, which scales linearly with respect to the vocabulary size. specifically, dark extracts verb and noun features separately, and relies on separate verb and noun knowledge graphs to predict unseen concepts before composing the action label. activity recognition is particularly well-suited for such a factorized approach, because nouns may be better captured using object detection-based approaches and verbs may be represented by motion. compared to prior work @xcite that use the same feature representation for both verb and noun, our separate features model noun and verb more precisely. in addition, we adopt disentanglement between the learned verb and noun features, so they compose more readily and improve generalization on unseen actions. though scalability is achieved using our factorized approach, verbs and nouns are actually not fully independent. for instance, the process of sanding an object and scrubbing an object are visually similar, however you are more likely to be scrubbing a car than sanding it. prior work @xcite models the verb-noun relationships by constructing quadratic compositional action nodes. in our model, when composing action labels, we take object affordances @xcite , namely the commonsense relationship of verbs and nouns, into consideration. we extract affordance knowledge from a caption corpus and build a scoring component to consider the relationship between verbs and nouns, to further improve the generalization ability. the basic idea of our proposed model is illustrated in figure 1 . furthermore, we investigate the evaluation of zero-shot compositional action recognition task and identify the drawback of existing metrics. with n v verbs, n n nouns, it constructs a n v × n n label space for possible actions. among these actions, some are invalid (e.g. peel a car) and some are valid but not presented in the dataset. in realworld applications, the model would need to make predictions in the whole n v × n n label space. but current evaluation protocols, implicitly or explicitly, only evaluate on compositional classes that are valid and presented in the dataset, which does not reflect the real difficulty of this task. we propose a new setting, where predictions are made and evaluated in the full n v × n n label space. the charades @xcite dataset is relatively small scale for testing zero-shot compositional action recognition @xcite . to promote further research, we propose a new benchmark based on the epic-kitchen @xcite ) dataset, which is an order of magnitude bigger both in number of classes and sample size. the key contributions of our paper are: ## method we propose dark -disentangled action recognition with knowledge-bases (figure 2 ). it extracts disentangled feature representations for verbs and nouns, then predicts classifier weights for unseen components using knowledge graphs, and com-poses them under object affordance priors. ## experiments in this section, we discuss experiment evaluation, setup and results. some implementation details are in appendix. foot_0 ## conclusion in this paper, we propose dark, a novel compositional action recognition model that reduces complexity from quadratic to linear, making the training more scalable. dark generalizes to unseen verb-noun pairs, and can be combined with knowledge bases to produce state-of-the-art compositional action recognition results.