Prompting Vision-Language Models For Aspect-Controlled Generation of Referring Expressions (2024)

Abstract: Referring Expression Generation (REG) is the task of generating a description that unambiguously identifies a given target in the scene. Different from Image Captioning (IC), REG requires learning fine-grained characteristics of not only the scene objects but also their surrounding context. Referring expressions are usually not singular; an object can often be uniquely referenced in numerous ways, for instance, by color, by location, or by relationship with other objects. Most prior works, however, have not explored this ‘aspect-based multiplicity’ of referring expressions. Hence, in this work, we focus on the Aspect-Controlled REG task, which requires generating a referring expression conditioned on the input aspect(s), where an aspect captures a style of reference. By changing the input aspect such as color, location, action etc., one can generate multiple distinct expressions per target region. To solve this new task, we first modify BLIP for aligning image-regions and text-expressions. We achieve this through a novel approach for feeding the input by drawing a bounding box around the target image-region and prompting the model to generate the referring expression. Our base REG model already beats all prior works in CIDEr score. To tackle Aspect-Controlled REG, we append ‘aspect tokens’ to the prompt and show that distinct expressions can be generated by just changing the prompt. Finally, to prove the high-quality and diversity of the data generated by our proposed aspect-controlled REG model, we also perform data-augmentation-based evaluation on the downstream Referring Expression Comprehension (REC) task. With just half of the real data augmented with the generated synthetic data, we achieve performance comparable to training with 100% of real data, using a SOTA REC model.

## methodology to summarize our full method pipeline, we build upon the blip multimodal mixture of encoder-decoder (med) model architecture @xcite , adapting it for aligning image-regions and text-expressions describing those regions. we introduce a novel and intuitive approach for feeding the input; by drawing a bounding box around the target region in the image and prompting the model to describe the marked region. to reinforce that the generated descriptions are unique referring expressions, we introduce a simple technique to craft negative examples that are utilized during contrastive learning. finally, we propose an intuitive yet novel approach for generating expressions conditioned on a given aspect (e.g. color, location etc.), by simply appending the aspect tag to the input prompt. to effectively evaluate our model, we generate synthetic data for training models for the downstream task of rec and use the rec performance as the evaluation metric. this evaluation approach allows handling multiple expressions (with various aspects) generated per target, by our reg model. the architecture and training setup of the model is shown in figure 2 . we follow the general structure of blip @xcite ) that consists of unimodal image and text encoder, an image-grounded text encoder and an image-grounded text decoder. our additions, here, are the modified image, the additional prompt input and new loss computations. the overall system is first pre-trained in a multitask manner, jointly minimizing region-expression contrastive loss, region-expression matching loss and expression generation loss. following this, image encoder and text decoder are fine-tuned only with the expression generation loss on larger images. in the following sections, we detail each of these new components of our proposed system. ## conclusion we present a model to generate referring expressions for a given object in arbitrary ways, where we use a prompt to guide our decoder. our approach, compared to traditional beam search, provides synthetic data of higher quality as evidenced in its diversity and ability to achieve higher accuracy with the same amount of training data.