CK -Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension (2023)

Abstract: The task of multimodal referring expression comprehension (REC), aiming at localizing an image region described by a natural language expression, has recently received increasing attention within the research comminity. In this paper, we specifically focus on referring expression comprehension with commonsense knowledge (KB-Ref), a task which typically requires reasoning beyond spatial, visual or semantic information. We propose a novel framework for Commonsense Knowledge Enhanced Transformers (CK-Transformer) which effectively integrates commonsense knowledge into the representations of objects in an image, facilitating identification of the target objects referred to by the expressions. We conduct extensive experiments on several benchmarks for the task of KB-Ref. Our results show that the proposed CK-Transformer achieves a new state of the art, with an absolute improvement of 3.14% accuracy over the existing state of the art.

## introduction referring expression comprehension (rec) aims at locating a target object/region in an image given a natural language expression as input. the nature of the task requires multi-modal reasoning and joint visual and language understanding. in the past few years, several rec tasks and datasets have been proposed, such as refcoco @xcite , refcocog @xcite and refcoco+ @xcite . these 'conventional' rec tasks typically focus on identifying an object based on visual or spatial information of the object, such as its colour, shape, location, etc.; therefore primarily evaluating a model's reasoning abilities over visual attributes and spatial relationships. in practice, however, people often describe an object using non-visual or spatial information -consider, for example, the sentence (expression) "give 1 the code will be available in https://github.com/ fightingfighting/ck-transformer me something soft but rich in starch to eat" @xcite . such instances require reasoning beyond spatial and visual attributes, and need to be interpreted with respect to the common sense knowledge (fact) embedded in the expressions, such as knowledge about which kind of objects are edible, soft and rich in starch in the given image. recently, @xcite proposed a new dataset, kb-ref, to evaluate the reasoning ability of a model over not only visual and spatial features but also commonsense knowledge. the dataset is devised such that at least one piece of fact from a knowledge base (kb) is required for a target object (referred to by an expression) to be identified. therefore, searching for appropriate facts from a kb is also crucial part in kb-ref. in contrast to the only existing work @xcite , in which for each object candidate, the top-k facts with the highest cosine similarity between the averaged word2vec @xcite embedding of the fact and the given expression are maintained, our framework focuses on multi-modal embedding and reasoning simultaneously over both the expression and the image to identify the top-k facts. multi-modal features encode richer information helping to improve reasoning over varying (semantic) contexts and identification of relevant facts; for example, the above example of expression can be answered with the object "banana" in an image (or, equivalently, with the object "mushed potato" in another image). in this paper, we propose a novel multi-modal framework for kb-ref -commonsense knowledge enhanced transformers (ck-transformer, ck-t for short) -that integrates (top-k) facts into all object candidates in an image for better identification of the target object. specifically, our contributions are four-fold: 1) we propose the ck-t (see figute 1) that effectively integrates diverse input from different modalities: vision, referring expressions and facts; 2) to the best of our knowledge, our approach is the first that introduces visual information into the identification of (top-k) relevant facts; 3) our approach achieves a new state of the art using only top-3 facts per (candidate) object, which is furthermore substantially more efficient compared to existing work utilizing as much as top-50 facts; 4) we introduce facts into 'conventional' rec tasks, leading to improved performance. ## related work referring expression comprehension with commonsense knowledge different from conventional rec tasks (see appendix a for details), kb-ref focuses on querying objects given an expression that requires commonsense knowledge reasoning. the authors benchmarked a baseline model, ecifa, for integration of facts, expression and image, and selects the target object by comparing the match scores between the image features and corresponding top-k fact features for all object candidates in the image @xcite . in our framework, we select top-k facts for each candidate by comparing the cosine similarity between the fact and expression embedding, where the embeddings are generated from a multi-modal encoder rather than a text encoder used in the ecifa model. pre-trained vision-language encoders several pre-trained multi-modal encoders @xcite @xcite have been proposed, achieving state-of-theart results on vision-language tasks. currently, uniter @xcite as one of powerful pre-trained encoders achieves the best performance on rec tasks (refcocos). in this paper, we adapt uniter such that it is used as a multi-modal en-coder in fact search and as part of the ck-t. ## methodology we formulate kb-ref as a classification problem based on an image i consisting of a set of candidates (image regions) @xmath0. specifically, given an expression e, an image i and a kb, we first search for top-k facts f k @xmath1, and then feed e, i, and f k i (the selected facts over i) into our ck-t simultaneously to predict the target object over all candidates. ## results we compare our ck-t to existing approaches on kb-ref task without and with facts. then we explore the importance of introducing visual information for fact search. furthermore, we introduce facts into the traditional refcocos dataset, which was collected from mscoco (lin et al., 2014) but differs in the types of expressions and object candidate settings. we extract image region features using an off-the-shelf detector, faster r-cnn with resnet-101 (ren et al., 2015), based on bounding boxes (bbxes) (ground-truth labels or predicted results from the detector). see appendix d and e for details about these datasets and experiment setting foot_0 . through parameter search on k and m (see figure 3 and 4 in appendix f), we keep @xmath2. ## analysis to investigate in what cases commonsense knowledge helps, we conduct a fine-grained analysis of model performance on the test set of kb-ref. specifically, we compare the samples predicted by model with and without facts (ck-t and ck-t-nf) on three aspects: object categories, spatial relationships and the size of the bounding box. object categories the test set contains 1502 categories and ck-t outperforms ck-t-nf on 1347 categories. top 10 categories for which most improvement is observed are shown in figure in case of the 155 categories that do not show improvement, we find that the average number of samples per category is 6.68, making the results less reliable. ## conclusion in this paper, we proposed ck-transformer, which effectively integrates commonsense knowledge and the expression into the representations of the corresponding visual objects for multi-modal reasoning on kb-ref. our ck-transformer achieves a new state-of-the-art performance on kb-ref using only top-3 most relevant facts. we also demonstrated that visual information is beneficial for fact search. finally, we show that commonsense knowledge improves conventional rec tasks across three different datasets.