Exploiting Commonsense Knowledge about Objects for Visual Activity Recognition (2023)

Abstract: Situation recognition is the task of recognizing the activity depictedin an image, including the people and objects involved. Previousmodels for this task typically train a classifier to identify theactivity using a backbone image feature extractor. We propose thatcommonsense knowledge about the objects depicted in an image can alsobe a valuable source of information for activity identification. Previous NLP research has argued that knowledge about the prototypicalfunctions of physical objects is important for language understanding,and NLP techniques have been developed to acquire this knowledge. Our work investigates whether this prototypical function knowledgecan also be beneficial for visual situation recognition. Webuild a framework that incorporates this type of commonsense knowledgein a transformer-based model that is trained to predict the actionverb for situation recognition. Our experimental results show thatadding prototypical function knowledge about physical objects doesimprove performance for the visual activity recognition task.

## introduction physical objects play an important role in our daily lives. people use different tools to achieve different goals in all kinds of situations. for example, we use a toothbrush to clean our teeth, a microwave oven to heat food, and a camera to take photos. the functions of physical objects is a type of commonsense knowledge that has been recognized to play an important role in natural language processing @xcite . physical objects play an important role in computer vision as well. there are well-established computer vision tasks that aim to identify the objects in an image, such as object detection @xcite and image classification @xcite . recently, attention has been paid to more comprehensive image understanding, such as identifying the salient event depicted in an image as well as relevant people and objects. situation recognition @xcite is the task of producing a structured summary of an image that describes the main activity and the entities that fill semantic roles for that activity. the task was originally defined using frame structures from framenet @xcite as the activity representation. for example, given the image shown in figure 1 , a system should identify a baking event (which is indexed in framenet as a type of cooking_creation activity), and recognize the corresponding semantic role/value pairs associated with framenet's cook-ing_creation frame. models for this task usually follow a two-step pipeline: (1) predict a verb that describes the activity depicted in the image, and (2) identify the entities associated with each semantic role. previous systems have relied solely on features extracted from the image and have not yet exploited any external commonsense knowledge. our work focuses on the activity recognition (verb prediction) part of the situation recognition task. we hypothesize that (a) correctly identifying the activity in an image strongly depends on recognizing the objects that appear in the image, and (b) explicit commonsense knowledge about physical objects can also be beneficial. more specifically, our work is motivated by recent research emphasizing the importance of commonsense knowledge about the prototypical functions of physical objects for language understanding @xcite . an intuitive extension to visual reasoning is that if an object appears in an image, especially when it is used by a person, the activity depicted in the image is likely to be the prototypical function associated with the object. for example, a woman holding a comb is probably brushing her hair, and a man holding a cookie sheet (as shown in figure 1 ) is probably baking. we explore these hypotheses by creating a transformer-based model that incorporates commonsense knowledge about the prototypical functions of physical objects for visual activity recognition. our experimental results confirm that correctly identifying the objects in an image is very important for activity recognition, and we show that providing explicit knowledge about the prototypical functions of objects can improve performance for this task. ## related work commonsense knowledge about physical objects has long been recognized to be important for natural language understanding @xcite . within the nlp community, a variety of recent projects have focused on acquiring and using different types of knowledge about physical objects, including relative physical knowledge @xcite , relative spatial relations @xcite , semantic plausibility @xcite , object affordances @xcite , and object usage status @xcite . the work most relevant to our research is @xcite , which developed a nlp method to learn the most typical way that people use humanmade physical artifacts. they used framenet frames as a representation for object functions and they created a dataset of physical objects paired with their prototypical function frames to evaluate their results. our research incorporates their prototypical function data into a transformer-based model for visual activity recognition. visual reasoning tasks, such as visual question answering @xcite and image captioning @xcite , have been widely explored for understanding images and videos. previous work has proposed to use external knowledge for visual tasks, such as image classification @xcite , object detection @xcite , and visual question answering @xcite . situation recognition is a task of recognizing the activity depicted in an image, including the people and objects involved in the activity and the roles these participants play. @xcite introduced the imsitu dataset, which associates images with a verb that describes the main action, and a set of semantic roles derived from framenet @xcite . they tackled this problem by first applying the vgg network @xcite to extract features from the image and then building a crf model to jointly predict the verb and semantic roles. several research efforts have further explored this task. @xcite used a graph neural network to capture the relations between semantic roles. @xcite used a lstm to jointly classify verbs and semantic roles. @xcite cast situation recognition as a query-based visual reasoning problem and further handled inter-dependencies between queries to overcome the sparsity issues of semantic roles. recently, @xcite proposed a collaborative framework using two transformer modules, and @xcite used contrastive learning to distinguish the correct activities from negative examples. all of these prior efforts have relied solely on features extracted directly from the image. our work aims to show that explicitly providing commonsense knowledge about objects can also be beneficial for visual activity recognition. ## methods given an image, the visual activity recognition task predicts a verb that describes the main activity in the image. figure 2 shows the framework of our model called arf (activity recognition with functions), which takes 3 sources of input: 1) the image, 2) nouns corresponding to the objects in the image, and 3) the names of framenet frames that describe the prototypical functions of the objects. we use the clip @xcite model, which has been pre-trained on both images and text, to generate an encoding for each of the 3 types of input. finally, we give the concatenated representation vectors as input to a transformer model that is trained to predict a verb for activity recognition. ## evaluation the imsitu data contains 126,102 images, with manually annotated activity verbs and frame structures. we follow the same data split @xcite as @xcite . we report verb prediction accuracy on both the development and test sets. when fine-tuning the transformer, we use batch size 32, hidden vector dimension 512, adamw optimizer with learning rate 1e-4 and train for 10 epochs. ## conclusion the prototypical functions of physical objects is a type of commonsense knowledge that is important for nlp. in this work, we showed that it can be a useful source of information for image understanding as well. specifically, we tackled the situation recognition task by building a transformer model that incorporates the functions of objects to predict the activity in an image. the experiments show that knowledge of the objects and their prototypical functions can improve performance on this task. however, automatically recognizing the objects in an image remains a challenge, and exploiting better object detection methods is an important direction for future work.