CLEVR -Implicit: A Diagnostic Dataset for Implicit Reasoning in Referring Expression Comprehension (2023)

Abstract: Recently, pre-trained vision-language (VL) models have achieved remarkable success in various cross-modal tasks, including referring expression comprehension (REC). These models are pre-trained on the large-scale image-text pairs to learn the alignment between words in textual descriptions and objects in the corresponding images and then fine-tuned on downstream tasks. However, the performance of VL models is hindered when dealing with implicit text, which describes objects through comparisons between two or more objects rather than explicitly mentioning them. This is because the models struggle to align the implicit text with the objects in the images. To address the challenge, we introduce CLEVR-Implicit, a dataset consisting of synthetic images and corresponding two types of implicit text for the REC task. Additionally, to enhance the performance of VL models on implicit text, we propose a method called Transforming Implicit text into Explicit text (TIE), which enables VL models to reason with the implicit text. TIE consists of two modules: (1) the prompt design module builds prompts for implicit text by adding masked tokens, and (2) the cloze procedure module fine-tunes the prompts by utilizing masked language modeling (MLM) to predict the explicit words with the implicit prompts. Experimental results on our dataset demonstrate a significant improvement of 37.94% in the performance of VL models on implicit text after employing our TIE method.

## introduction in recent years, pre-trained vision-language models have demonstrated remarkable achievements in various vision-language downstream tasks @xcite @xcite @xcite , including referring expression comprehension (rec). rec involves identifying a specific object in an image based on corresponding textual descriptions. a common approach is to pre-train a vision-language model on large-scale image-text pairs to learn the alignment between words in textual descriptions and objects in the images. fine-tuning is then performed to optimize model performance @xcite @xcite . despite the state-of-the-art results demonstrated by these methods, they still exhibit limitations. pretrained models are good at processing explicit text, which describes the appearance attributes of the objects in the image, whether they are reference or target objects. for example, consider the phrase "the object left on the red cube.", models rely on the explicit word "red cube" to locate the referred object in the image. however, they encounter challenges when it comes to implicit reasoning, which is processing the implicit text in rec. implicit reasoning occurs when the textual expression indirectly describes an object through a comparison between two or more objects, one type of implicit text is formatted like "the left one of two same color objects". the phrase "two same color objects" does not describe the appearance attributes of any objects, only the relationships between them, and there is no direct reference for the models to infer the target. consequently, models struggle to align "two same color objects" with any objects in the image. to evaluate the performance gap between explicit and implicit text, we conducted experiments using existing methods on both types of text separately. preliminary results indicate that the performance on implicit text is approximately 42% lower compared to explicit text. besides, the implicit text also plays a significant role in the domain of human-computer interaction. however, existing referring expression datasets, such as refcoco @xcite , refcoco+ @xcite , refcocog @xcite , and clevr-ref+ @xcite , primarily consist of explicit text. unfortunately, there is currently no dataset available that is specifically designed to provide research data on implicit text. to address the lack of research data and methods for implicit text, we propose a synthetic dataset called clevr-implicit, specifically for implicit reasoning in rec. unlike existing datasets, the implicit text we constructed is a subset of all types of implicit text, since they describe synthetic images, and do not need to consider external knowledge. specifically, we focus on the more difficult types (only containing relationships between objects without any description of their appearance attributes). we evaluate the current pre-trained rec models on our clevr-implicit dataset with these types of implicit text. in real-world scenarios, most humans will first extract explicit information from visual context and text information when processing these types of implicit text, and then reason the target object from the extracted explicit information. inspired by the thinking mode of humans, we simulate the process of humans reasoning about the implicit text and introduce a method called tie (transform the implicit text into explicit text), which enables pre-trained models to process the implicit text by extracting the explicit information. by employing tie, rec models facilitate alignment between explicit words and corresponding images on clevrwe summarize our contributions as follows: ## methodology this section begins by explaining the construction process of the clevr-implicit dataset. subsequently, we introduce our proposed tie method, which involves converting the implicit text within the clevr-implicit dataset into explicit text. this explicit text is then utilized in the referring expression comprehension task through the fine-tuning process of pre-trained vision-language models. ## implicit text in real-world scenarios the implicit descriptions in real-world scenarios are more complex and difficult because they not only describe the targets implicitly through the geometrical attributes of color, material, etc., but also contain more attributes as well as external factors. ## conclusion in this paper, we analyze the significance of implicit reasoning, along with the lack of investigation in this domain. to address this, we introduce the clevr-implicit dataset for implicit reasoning in the rec task. this dataset utilizes synthesized images to generate two types of implicit text: "same" and "different." evaluating the performance of existing pre-trained models on clevr-implicit, we observe model confusion when attempting to align implicit text with images. to overcome this challenge, we propose the transform implicit text to explicit text (tie) method, which converts implicit text into explicit text. our experiments demonstrate that the reasoning capability of the models improves significantly through the alignment of explicit words and images.