F ine C ops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension (2024)

Abstract: Referring Expression Comprehension (REC) is a crucial cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding. Consequently, it serves as an ideal testing ground for Multi-modal Large Language Models (MLLMs). In pursuit of this goal, we have established a new REC dataset characterized by two key features: Firstly, it is designed with controllable varying levels of difficulty, necessitating multi-level fine-grained reasoning across object categories, attributes, and multi-hop relationships. Secondly, it includes negative text and images created through fine-grained editing and generation based on existing data, thereby testing the model’s ability to correctly reject scenarios where the target object is not visible in the image—an essential aspect often overlooked in existing datasets and approaches. Utilizing this high-quality dataset, we conducted comprehensive evaluations of both state-of-the-art specialist models and MLLMs. Our findings indicate that there remains a significant gap in achieving satisfactory grounding performance. We anticipate that our dataset will inspire new approaches to enhance visual reasoning and develop more advanced cross-modal interaction strategies, ultimately unlocking the full potential of MLLMs.

## introduction despite significant advancements in multimodal large language models (mllms), a critical challenge remains in ensuring these models' responses are grounded in visual content rather than solely derived from linguistic cues @xcite @xcite . vision-language models (vlms) often treat language as a bag of * corresponding author. current evaluation methods utilize visual question answering or image-text retrieval to evaluate the compositional reasoning or grounding abilities of mllms. however, these methods provide an indirect assessment of the models' visual grounding capabilities. in contrast, the referring expression comprehension (rec) task requires a model to directly output the bounding box coordinates of a target object based on a given language expression, serving as an ideal testing ground for mllms. recent mllms, leveraging substantial grounding data @xcite and specifically designed visual modules @xcite , have achieved impressive results on common rec benchmarks like refcoco/+/g @xcite . however, these benchmarks lack considerations of compositional reasoning, allowing models to perform well without understanding linguistic structure or even without the expression @xcite . additionally, current vlms struggle with negative samples, where the target object is absent from the image @xcite @xcite . this limitation is further exacerbated by the lack of robustness in existing datasets, which fail to provide the necessary complexity and variability to thoroughly evaluate mllms. in response, we introduce finecops-ref, a benchmark specifically designed to address these limitations. our dataset introduces controlled difficulty levels, compelling mllms to perform fine-grained reasoning across object categories, attributes, and multi-hop relationships. we classify the difficulty levels based on the number of at-tributes and relationships necessary for locating the target object. for instance, if there is only one possible target in the image, the difficulty level is 1 regardless the complexity of the expression. if the model needs to understand at least two or more relationships and attribute information, the difficulty level is 3. moreover, finecops-ref incorporates negative samples crafted through meticulous editing, testing the models' resilience against misalignments and hallucinations, thereby assessing their true visual grounding capabilities. our comprehensive evaluation with state-of-theart models reveals a significant gap in grounding performance, highlighting the need for advanced visual reasoning strategies. we present several core findings in our study. firstly, for simple rec tasks with a difficulty level 1, traditional vision-language models, despite their relatively smaller parameter sizes, maintained a significant advantage. secondly, all models exhibited poorer performance at difficulty levels greater than 1, while mllms demonstrated stronger capabilities under these conditions. in terms of negative data, all models showed weak performance, even in the simplest scenarios where the image does not contain an object matching the category specified in the expression. additionally, we observed a positive correlation between precision on positive samples and recall with negative samples, with traditional vision-language models and mllms displaying different tendencies. to enhance the fine-grained compositional reasoning capabilities of existing models, we employed the same pipeline used to construct our benchmark to create a rich training dataset that includes both positive and negative samples. finetuning on this training dataset significantly improved model performance, with further improvements observed on the refcoco/+/g dataset. we make finecops-ref and the code for our data generation pipeline publicly available under the cc by 4.0 license. ## related works referring expression comprehension. the rec methods can generally be divided into two categories based on whether or not it uses llms: specialist and mllms. specialists typically extract text and image features separately and perform multi-stage fusion @xcite @xcite . their training tasks often include various object location tasks. recently, @xcite achieved excellent results on two visual grounding (vg) benchmarks by leveraging hard negative samples in training. on the other hand, mllms directly input the projected visual features into the llm. recent methods aim to enhance grounding capabilities in mllms through dataset construction with coordinate information and additional visual modules. common methods for datasets include transforming traditional visual datasets into an instruction-following format using templates @xcite @xcite , correlating object coordinates with existing captions @xcite , and using llms to generate grounded questionanswer pairs based on images, object coordinates, and captions @xcite . in terms of visual modules, some methods integrate additional visual components, such as glamm @xcite and llava-grounding @xcite , while others extract regional features as additional inputs @xcite @xcite . evaluation of compositional reasoning. current multimodal models, including advanced mllms like gpt-4v, exhibit poor compositional reasoning, often treating language as a bag of words without considering word order, attributes, or relationships between objects @xcite @xcite @xcite . common evaluation benchmarks involve constructing hard negative captions to test models' capabilities, such as distinguishing between "a mug in some grass" and "some grass in a mug" @xcite @xcite . @xcite found that previous benchmarks have language biases and that a simple grammar model can distinguish negative captions. some benchmarks focus on negative images @xcite @xcite , while others primarily focus on spatial relationships @xcite @xcite . for rec tasks, @xcite critically examined refcocog, showing that 83.7% of test instances do not require reasoning on linguistic structure, and proposed the ref-adv dataset, which perturbs original expressions to refer to different target objects. clevr-ref+ @xcite is a synthetic dataset emphasizing relationships, ## finecops-ref finecops-ref includes both positive and negative data. figure 1 illustrates the dataset construction pipeline. ## conclusion in this work, we introduced finecops-ref, a novel dataset for fine-grained compositional referring expression comprehension with varying difficulty levels and negative samples. our evaluations reveal that while current mllms perform well on traditional rec benchmarks, they struggle with advanced compositional reasoning and accurate rejection of negative samples. our dataset provides strong support for the evaluation of the model's compositional grounding ability, and the training set can also serve as a good supplement to existing training data for compositional rec. we hope finecops-ref can inspire further research into enhancing compositional visual grounding.