Scene Graph Enhanced Pseudo-Labeling for Referring Expression Comprehension (2023)

Abstract: Referring Expression Comprehension (ReC) is a task that involves localizing objects in images based on natural language expressions. Most ReC methods typically approach the task as a supervised learning problem. However, the need for costly annotations, such as clear image-text pairs or region-text pairs, hinders the scalability of existing approaches. In this work, we propose a novel scene graph-based framework that automatically generates high-quality pseudo region-query pairs. Our method harnesses scene graphs to capture the relationships between objects in images and generate expressions enriched with relation information. To ensure accurate mapping between visual regions and text, we introduce an external module that employs a calibration algorithm to filter out ambiguous queries. Additionally, we employ a rewriter module to enhance the diversity of our generated pseudo queries through rewriting. Extensive experiments demonstrate that our method outperforms previous pseudo-labeling methods by about 10%, 12%, and 11% on RefCOCO, RefCOCO+, and RefCOCOg, respectively. Furthermore, it surpasses the state-of-the-art unsupervised approach by more than 15% on the RefCOCO dataset.

## introduction referring expression comprehension (rec) is an important visual-linguistic task that aims to locate objects in images based on given textual referring expressions. referring expressions, such as "give me that apple in the basket," are commonly used in social communication. the ability to accurately comprehend such expressions in real-world scenarios is also essential for robots and other intelligent agents to enable natural interactions @xcite . in recent years, considerable progress has been made in rec task, particularly for fully-supervised models @xcite ; * *corresponding author original query from refcoco dataset: the bowl that has blueberries on it pseudo-q query: white bowl white bowl sgepg query: corner left top bowl with blueberry above table on the right side, there's a white bowl with green broccoli inside figure 1 : example of queries generated by pseudo-q and sgepg methods, based on two different objects in an image from the refcoco dataset. note that the query generated by sgepg without rewriting is shown in red, while the query generated by sgepg with rewriting is shown in blue. @xcite @xcite trained on manually annotated region-query data. however, constructing large-scale rec datasets, especially for textual queries referring to the objects, is rather time-consuming. to obtain the clear textual queries, annotators need to provide specific visual information about the referred object, such as its category and attributes. moreover, due to the presence of multiple similar and confusing objects in images, queries often require additional details to distinguish the referred object, such as spatial information (left, right), object state (running, jumping), and object relationships (e.g., person holding ball). although some weakly-supervised methods @xcite @xcite partially alleviate the annotation burden, they still rely on paired image-query data, limiting their applicability in the real-world scenes. to address the annotation challenges, several works @xcite @xcite have explored unsupervised approaches that do not rely on paired annotations. nonetheless, these approaches either employ statistical hypothesis testing, make simple assumptions, or only investigate the shallow relation between objects, leading to poor performance in complex scenes. for instance, pseudo-q @xcite , which adopts a method of generating pseudo-labels to train a supervised model. it utilizes an offline object detector to extract salient objects from images and generates pseudo queries using predefined templates with object labels and attributes. however, pseudo-q has two limitations: (1) it fails to capture deep relation information among objects in images, leading to pseudo queries that only partially cover the semantic space of referring expressions and deviate significantly from human expressions. (2) the absence of a validation mechanism for the generated pseudo queries introduces the possibility of producing incorrect instances. both limitations can lead to a degradation in model performance. in this paper, we present sgepg (scene graph enhanced pseudo-query generation), an unsupervised framework designed to generate high-quality region-query pairs. sgepg leverages the generated scene graph to extract rich semantic information about objects and capture their complex relationships, enabling the selective generation of region-query pairs from the input image. we next introduce a straightforward yet effective strategy based on the scene graph, which helps filter out ambiguous queries. it is crucial to address ambiguity as unclear region-text pairs can yield incorrect supervisory signals, which ultimately degrade the performance of the rec model. additionally, we propose a module to rewrite some of the generated queries. this rewriting process enhances the diversity of the generated pseudo queries while also making them more akin to human expressions. as depicted in figure 1 , compared to pseudo-q, the sgepg method generates pseudo queries that are both unambiguous and exhibit higher quality. moreover, the generated queries by sgepg also demonstrate more complex inference paths, such as the paths from "bowl" to the "blueberries". specifically, the sgepg framework comprises four essential components: core scene graph generation, which is responsible for obtaining an informative core graph to capture object relations; non-ambiguous query generation, which focuses on generating unambiguous queries; diversity driven query rewriter, which is designed to rewrite some of the generated queries, aiming to increase the diversity and alleviate the differences between pseudo-queries and human expressions; visual-language module, which aims to capture the contextual relationship between the image regions and corresponding pseudo queries. the experimental results demonstrate the effectiveness of our method, which significantly reduces the manual labeling cost by about 40% on the refcoco dataset, with only a slight decrease in performance compared to the fully supervised setting. in addition, our proposed method consistently outperforms the state-of-the-art unsupervised approach by about 10-13 points on the refcoco @xcite , refcoco+ @xcite , and refcocog @xcite datasets. our main contributions can be summarized as follows: ## methodology in this section, we present the sgepg framework, which aims to automatically generate regionquery pairs from given images and train models to tackle rec task using these pseudo labels. as depicted in figure 2 , the sgepg framework consists of four key components: core scene graph generation, which is responsible for obtaining an informative core graph to capture object relations; non-ambiguous query generation, which focuses on generating unambiguous queries; diversity driven query rewriter, which is designed to rewrite some of the generated queries, aiming to increase the diversity and alleviate the differences between pseudo-queries and human expressions; visual-language module, which aims to capture the contextual relationship between the image re-gions and corresponding pseudo queries. note that in this work, we mainly focus on the first three modules, with the objective of generating pseudo-labels of good quality for training visual-language models designed for the rec task. to accommodate space limitations, certain methodological details may have been omitted. for a more comprehensive understanding, please refer to appendix a for detailed information regarding the methods employed in this study. ## conclusion in this paper, we propose sgepg, a novel pseudolabeling rec method that leverages scene graphs to improve the generation of pseudo queries. by utilizing the scene graph, sgepg captures object relationships and generates pseudo queries that are rich in relation information, thereby improving the training of rec models. additionally, we propose a simple yet effective strategy to filter out ambiguous generated queries. our method outperforms previous pseudo-labeling approaches on three benchmark datasets and achieves comparable or superior performance compared to other unsupervised rec methods. furthermore, sgepg offers the advantage of significantly reducing manual labeling costs while maintaining the model's performance.