RECANTF ormer: Referring Expression Comprehension with Varying Numbers of Targets (2024)

Abstract: The Generalized Referring Expression Comprehension (GREC) task extends classic REC by generating image bounding boxes for objects referred to in natural language expressions, which may indicate zero, one, or multiple targets. This generalization enhances the practicality of REC models for diverse real-world applications. However, the presence of varying numbers of targets in samples makes GREC a more complex task, both in terms of training supervision and final prediction selection strategy. Addressing these challenges, we introduce RECANTFormer, a one-stage method for GREC that combines a decoder-free (encoder-only) transformer architecture with DETR-like Hungarian matching. Our approach consistently outperforms baselines by significant margins in three GREC datasets.

## introduction referring expression comprehension (rec) focuses on generating an image bounding box tightly encompassing a region referred to by a natural language query. this is a core task in multi-modal information processing with potential to influence a wide range of applications including instruction following robots @xcite , situated multi-modal dialogues @xcite , and interactive photo editing @xcite . despite advances in rec on datasets like refcoco/+/g @xcite current methods assume that a single referring expression always refers to a single object instance in the image. this simplification limits their real-world applicability, as they cannot handle expressions with no or multiple matching instances. several datasets; generalized rec @xcite , visual query detection @xcite , and ref-zom @xcite have been proposed to bridge this gap between real-world and classic rec datasets. expressions in these datasets may refer to zero, one, or many instances in an image. (in this work, we adopt the term 'grec' coined by @xcite to refer to this family of tasks). despite the existence of suggested datasets and their corresponding baselines based on state-ofthe-art classic rec models, no models have been developed specifically addressing rec with varying numbers of targets, including zero targets. the grec task is more challenging than classic rec, where an image with n referable objects results in n distinct targets in classic rec, as opposed to 2 n distinct combinations of objects in grec. unlike classic rec datasets, which are susceptible to models exploiting biases @xcite the large pool of possible distinct combinations in the grec task makes it extremely difficult for models to exploit biases. furthermore, a top-1 selection strategy (or top-k variant) which is prevalent in classic rec models with the one-to-one assumption @xcite , is unsuitable when there are varying numbers of targets. a confidence-score threshold-based selection strategy is a viable alternative. however, we demonstrate that the threshold-based approach also leads to a significant drop in performance when current rec models are trained and evaluated on grec datasets. to overcome these limitations in classic rec methods, we introduce recantformer: a transformer-based framework for referring expression comprehension with varying number of targets. to address the challenge of training the model with a varying number of targets, we leverage hungarian matching loss similar to detr @xcite , where bipartite matching is calculated between the set of proposed boxes and ground truth boxes. however, differing from detr, which is a transformer encoder-decoder based approach for object detection, recant-former only employs transformer encoders with simple mlp-based prediction heads. inspired by the success of using a separate token for grounding @xcite , to allow for multiple potential targets in grec, we propose a multimodal transformer encoder with multiple learnable localization tokens. our selection of an decoder-free architecture is driven by the training-inefficient nature of encoder-decoder based detr-like architectures, as shown in @xcite . ## conclusion this paper presents recantformer, the first framework focused on the challenging task of generalized referring expression comprehension (grec). recantformer has a simple, decoderfree transformer-based architecture and demands minimal visual-language training data. our model effectively utilizes the powerful multimodal fusion capabilities of transformers encoders to outperform grec benchmarks across 3 datasets. by effectively handling referring expressions with a varying number of target objects, including no-target scenarios, recantformer expands the range of applications for rec.