Interactive Text-to-Image Retrieval with Large Language Models: A Plug-and-Play Approach (2024)

Abstract: In this paper, we primarily address the issue of dialogue-form context query within the interactive text-to-image retrieval task. Our methodology, PlugIR, actively utilizes the general instruction-following capability of LLMs in two ways. First, by reformulating the dialogue-form context, we eliminate the necessity of fine-tuning a retrieval model on existing visual dialogue data, thereby enabling the use of any arbitrary black-box model. Second, we construct the LLM questioner to generate non-redundant questions about the attributes of the target image, based on the information of retrieval candidate images in the current context. This approach mitigates the issues of noisiness and redundancy in the generated questions. Beyond our methodology, we propose a novel evaluation metric, Best log Rank Integral (BRI), for a comprehensive assessment of the interactive retrieval system. PlugIR demonstrates superior performance compared to both zero-shot and fine-tuned baselines in various benchmarks. Additionally, the two methodologies comprising PlugIR can be flexibly applied together or separately in various situations.

## introduction text-to-image retrieval, a task focused on locating target images in an image database that correspond to an input text query, has seen significant advancements thanks to the development of vision-language multimodal models @xcite . conventionally, methods in this domain have adopted a single-turn retrieval approach, reliant on the initial text input, which necessitates comprehensive and detailed descriptions from users. recently, @xcite have introduced a chat-based image retrieval system utiliz-ing large language models (llms) @xcite as questioners to facilitate multi-turn dialogues, enhancing retrieval efficiency and performance even with simplistic initial image descriptions given by users. however, this chat-based retrieval framework confronts certain limitations, including the requirement of fine-tuning to adeptly encode dialogue-style texts, a process that is both resource-intensive and impractical for scalability. moreover, the reliance of the llm questioner on initial descriptions and dialogue histories, without the ability to view the image candidates, poses a risk of generating queries about non-existent attributes in the target image, based on the llm's parametric knowledge. to overcome these challenges, this paper introduces plugir, a novel plug-and-play interactive text-to-image retrieval methodology that is tightly coupled with llms. plugir comprises two key components: context reformulation and context-aware dialogue generation. harnessing the instruction-following proficiency of llms, plugir reformulates the interaction context between users and questioners into a compatible format for pretrained vision-language models @xcite . this process enables the direct application of an array of multimodal retrieval models, including black-box variants, without necessitating further fine-tuning. additionally, our approach ensures that the llm questioner's inquiries are grounded in the context of the retrieval candidates set, thereby allowing it to formulate questions pertinent to the target image's attributes. during this process, we inject the retrieval context in text form as an input context for the llm questioner to reference. subsequently, our methodology also incorporates a filtering process that selects the most contextaligned, non-repetitive questions, thereby streamlining the search options. figure 1 illustrates the overall structure of our proposed interactive textto-image retrieval system. we identify three critical aspects for assessing interactive retrieval systems: user satisfaction, efficiency, and ranking improvement significance. we show that existing metrics, such as recall@k and hits@k @xcite , fall short in these areas. for instance, hits@k fails to account for efficiency, which is better with fewer interactions to locate the target image. to resolve these issues, we introduce the best log rank integral (bri) metric. bri effectively covers all three essential aspects, offering a comprehensive evaluation independent of a specific rank k, unlike recall@k or hits@k. we empirically demonstrate that bri aligns more closely with human evaluation compared to existing metrics. experiments across diverse datasets, including visdial @xcite , coco @xcite , and flickr30k @xcite , show that plu-gir significantly outperforms the existing interactive retrieval systems using zero-shot or fine-tuned models @xcite . moreover, our approach shows significant adaptability when applied to diverse retrieval models, including black-box models. this compatibility extends the utility of our approach, allowing it to be adapted to a broader spectrum of applications and scenarios. we summarize our contributions as follows: vision-language models vision-language models (vlms) have emerged as a pivotal area in ai research, aiming to bridge the gap between visual and textual understanding. clip @xcite introduces the vision-language landscape by leveraging a contrastive learning framework. by jointly embedding images and their associated text descriptions, clip demonstrates a robust ability to perform various vision-language tasks, especially in zero-shot classification across a wide range of concepts. subsequently proposed blip @xcite goes further by introducing a model capa-ble of both understanding and generation task, and addressing the issue of noisy captions in web data used for pre-training. as a result, blip exhibits exceptional performance in zero-shot image-text retrieval. several pioneering models, such as blip-2 @xcite , have significantly advanced this field, exhibiting remarkable capabilities in crossmodal representation learning, and we make use of these vlms as our text-to-image retrieval models. large language models beginning with the generative pretrained transformer (gpt) series @xcite @xcite , a variety of works have proposed scaling up the parameters of language models to the billion-scale @xcite . increasing the number of parameters has not only enhanced the performance of language models but also revealed various emergent abilities @xcite , which have enabled remarkable performance in a range of downstream tasks, including zero-shot and few-shot learning. beyond training high-performance llms, topics on techniques like chain-of-thought prompting @xcite and self-consistency @xcite to effectively extract answers from trained llms are active research area. ## conclusion we investigate the dialogue-form interaction with llms in the context of text-to-image retrieval. our proposed plugir progressively refines the text query for image retrieval using the dialogue between llm questioner and the user. specifically, an llm converts the dialogue into a format better understood by retrieval models. plugir enables the direct application of an array of multimodal retrieval models, including black-box models, without necessitating further fine-tuning. moreover, we newly propose the best log rank integral (bri) metric allowing for the measurement of comprehensive performance in multi-turn tasks. we verify the effectiveness of our retrieval system across a diverse range of environments, highlighting its versatile plug-and-play capabilities.