Segment-Level and Category-Oriented Network for Knowledge-Based Referring Expression Comprehension (2023)

Abstract: Knowledge-based referring expression comprehension (KB-REC) aims to identify visual objects referred to by expressions that incorporate knowledge. Existing methods employ sentence-level retrieval and fusion methods, which may lead to issues of similarity bias and interference from irrelevant information in unstructured knowledge sentences. To address these limitations, we propose a segment-level and category-oriented network (SLCO). Our approach includes a segment-level and prompt-based knowledge retrieval method to mitigate the similarity bias problem and a category-based grounding method to alleviate interference from irrelevant information in knowledge sentences. Experimental results show that our SLCO can eliminate interference and improve the overall performance of the KB-REC task.

## introduction referring expression comprehension (rec), a.k.a. visual grounding, aims to identify a visual object referred to by a referring expression that disambiguates multiple objects @xcite . as a core task of languagevision fields, rec benefits many downstream multimodal tasks, e.g., robotics @xcite and vision-and-language navigation @xcite . to explore a broader domain of knowledge, wang et al. extend the rec task to knowledgebased referring expression comprehension (kb-rec), and propose a baseline model and benchmark @xcite . this task requires the use of external knowledge (e.g., commonsense and encyclopedia) to refer to objects. this necessitates the model's ability to retrieve knowledge related to expressions and associate it with image and expression, enabling localization of the referent. ## proposed method an illustration in fig. 2 , slco contains three main modules: (1) a segment detection module, which identifies knowledge segments and visual segments in an expression; (2) a prompt-based retrieval module, which employs knowledge segments as hints to elicit knowledge category from language models; and (3) a category-based grounding module, which associates knowledge category and visual segments for object localization. ## conclusion in this paper, we propose a segment-level and category-oriented network to endow the model with the ability to identify and utilize the knowledge and visual segments in a targeted manner. specifically, the proposed method uses knowledge segments to retrieve knowledge, which addresses the similarity bias problem in the sentence-level method. additionally, our category-oriented retrieval method can elicit knowledge categories from language models, mitigating the interference from irrelevant information in knowledge sentences. experimental results demonstrate the effectiveness of the proposed method in addressing two limitations of the existing methods, thus improving the accuracy of the kb-rec task. in future work, we will explore more fine-grained information in expressions and combine it with knowledge and visual content.