Find Someone Who: Visual Commonsense Understanding in Human-Centric Grounding (2022)

Abstract: From a visual scene containing multiple people, human is able to distinguish each individual given the context descriptions about what happened before, their mental/physical states or intentions, etc. Above ability heavily relies on human-centric commonsense knowledge and reasoning. For example, if asked to identify the “person who needs healing” in an image, we need to first know that they usually have injuries or suffering expressions, then find the corresponding visual clues before finally grounding the person. We present a new commonsense task, Human-centric Commonsense Grounding, that tests the models’ ability to ground individuals given the context descriptions about what happened before, and their mental/physical states or intentions. We further create a benchmark, HumanCog, a dataset with 130k grounded commonsensical descriptions annotated on 67k images, covering diverse types of commonsense and visual scenes. We set up a context-object-aware method as a strong baseline that outperforms previous pre-trained and non-pretrained models. Further analysis demonstrates that rich visual commonsense and powerful integration of multi-modal commonsense are essential, which sheds light on future works. Data and code will be available at https://github.com/Hxyou/HumanCog.

## introduction visual scenes often involve multiple people. for instance, as in movies, a frame can involve multiple characters. complex human interaction happens because different people may have different intentions, roles, and emotions. when observing such scenes, humans can understand the scene and differentiate the characters from each other according to the context description, such as what will happen/happened to them, attributes, mental/physical states, and intentions. ## related work visual grounding dataset ground the corresponding regions in images given text information is an essential task to bridge image and text modalities. there are in general two conventional settings in existing grounding datasets. in the first setting, the entire sentence is mainly describing one object and its environment/attribute, and only refers to one box in image, which is termed as referring expression comprehension (rec). refcoco @xcite , refcoco+ @xcite and re-fcocog @xcite are commonly used rec datasets annotated on top of mscoco @xcite . refcoco and refcoco+ are annotated using referit game @xcite , but refcoco+ focuses more on appearance description since location words are not al-lowed. refcocog is collected by mechanical turks in a non-interactive setting. clevr-ref+ @xcite is collected on synthesized images where objects of different attributes are put on plane. kb-ref @xcite enriches the sentence description by injecting knowledge retrieved from the external knowledge base. flick30k entity @xcite is the pioneer to establish the second setting: multiple phrases inside one caption can be grounded to different boxes. who's waldo @xcite further studies grounding the persons in sentence. our work differs from who's waldo and kb-ref in that the descriptions contain rich human-centric commonsense such as temporal, causal, mental, etc. visual grounding approaches current methods can be divided into two categories: one-stage and two-stage @xcite . in two-stage methods, a set of candidate object regions are first detected by object detection models, then multimodal models are used to predict the links between detected boxes and text. lstm-based models @xcite , attention-based models @xcite @xcite , graph-based models @xcite and pre-trained models @xcite @xcite are explored. in one-stage models, the coordinates of grounded object box are directly predicted by a single model @xcite @xcite .more can be found in a survey @xcite . since our dataset already gives ground-truth person candidate boxes, i.e., the first stage results in two-stage schema are provided, we mainly focuses on building better model for image-text understanding (second stage in two-stage schema). multimodal commonsense reasoning multimodal commonsense reasoning has attracted wide research interest in recent years. vcr @xcite introduces commonsense question that requires a deep understanding of both image and text, and is formulated as a multi-choice answering task. visualcomet @xcite focuses on inferring the temporal and causal information given current image and description, regarded as a generation task. vlep @xcite also requires machine to predict future event but is in multi-choice answering format. we are similar to vcr in that we collect images from vcr and transform the questions&answers in vcr to statements. however, we differs from above works in that we target at the human-centric commonsense grounding ability of machines. ## task: human-centric commonsense grounding we present a challenging task, human-centric commonsense grounding, to mimic the inference ability of humans to distinguish wanted persons in image by corresponding commonsensical description. the input of one sample in this task includes: (1) an image i. (2) a set of n (@xmath0, covering all the persons in the image. given above input, the goal of the task is to ground/refer the person links to corresponding correct person boxes out of all candidate person boxes, i.e., arg max r f (t i |t, i, r) = r j , {t i , r j } ∈ l, where l is the set of ground-truth reference pairs and f is the desired model. we evaluate the accuracy of correct prediction among candidate person boxes. take fig. 1 as an example. for the top picture, a commonsensial description is "personx feels distressed", where "personx" is an person link, and its corresponding ground-truth person in image should be "person2". it's noted that there might be more than one person links referring to the same person in image (see the bottom picture in fig. 1 ). ## dataset collection and analysis the humancog dataset contains 130k commonsensical descriptions on 67k images, where in total 230k persons are grounded. in the following, we describe how humancog is constructed and annotated, and provide detailed analysis of the dataset. ## method in this section, we introduce a context-object-aware method as a strong baseline to solve the task. to jointly encode vision and text input, our architecture is built on a pre-trained vision-language transformer, uniter @xcite for its generality, which will be covered in sec. 5.1. in sec. 5.2, we introduce the classification loss and the proposed context contrastive loss that can facilitate the visual commonsense learning between the human-object interaction. the diagram of method is shown in fig. 6 5.1 architecture visual input given the image i and candidate person boxes r, we follow uniter @xcite to use the faster r-cnn @xcite to extract the pooled roi features of each region r i as visual features. location features are encoded by a 7-dimensional vector, [x 1 , y 1 , x 2 , y 2 , w, h, w * h] 1 . visual and location features are transformed into the same dimension through two fc layers, and are then summed up and normalized by a ln, as the input features of each region. additionally, to complement person representation and enrich the visual scene understanding, we take extra detected object proposals 2 r ′ by faster r-cnn and append their features together with candidate person boxes as input. it's validated in experiments that the additional proposals are essentially helpful to this task. transformer encoder the visual and textual features are input into the transformer @xcite , pre-trained with 9.5m image-caption pairs. the self-attention layers inside enable the contextualization of the two modalities. we take the hidden layers' features for loss calculation. ## results for evaluation, we comprehensively study the effect of various methods on this task. we first introduce previous state-of-the-art (sota) methods and compare them with the proposed context-objectaware model (sec. 6.2). and we conduct ablation experiments to quantify the importance of different components in this task (sec. 6.3). further, we provide qualitative results for analysis (sec. 6.4). ## conclusion in this work, we present a new task: human-centric commonsense grounding, where machines are required to ground the persons mentioned in a commonsensical description. correspondingly, we collect a new dataset for training and evaluation. also, we proposed a context-object-aware method that exploits background context objects via context contrastive loss for a strong vision-language understanding. through detailed analysis, we find there is still an ample room for improvement and we point out the potential directions for further works.