Balancing Visual Context Understanding in Dialogue for Image Retrieval (2024)

Abstract: In the realm of dialogue-to-image retrieval, the primary challenge is to fetch images from a pre-compiled database that accurately reflect the intent embedded within the dialogue history. Existing methods often overemphasize inter-modal alignment, neglecting the nuanced nature of conversational context. Dialogue histories are frequently cluttered with redundant information and often lack direct image descriptions, leading to a substantial disconnect between conversational content and visual representation. This study introduces VCU, a novel framework designed to enhance the comprehension of dialogue history and improve cross-modal matching for image retrieval. VCU leverages large language models (LLMs) to perform a two-step extraction process. It generates precise image-related descriptions from dialogues, while also enhancing visual representation by utilizing object-list texts associated with images. Additionally, auxiliary query collections are constructed to balance the matching process, thereby reducing bias in similarity computations. Experimental results demonstrate that VCU significantly outperforms baseline methods in dialogue-to-image retrieval tasks, highlighting its potential for practical application and effectiveness in bridging the gap between dialogue context and visual content.

## introduction recent advancements in communication technology have significantly enhanced online conversational systems @xcite @xcite , which now play a crucial role in facilitating instant messaging and information sharing. however, purely textual exchanges often fail to fully convey the speaker's intentions and emotions, leading to the development of multimodal conversational systems that enrich dialogue with images and audio etc (liao * corresponding author. et @xcite @xcite @xcite @xcite . as shown in figure 1 , sharing photographs provides a vivid and intuitive means of communication, making image-sharing capabilities essential in dialogue systems. traditional text-to-image retrieval techniques @xcite @xcite , which rely on direct image categories or detailed descriptions, fall short in dialogue contexts. the primary challenge is to select relevant images from a predefined repository based on the ongoing conversation, which involves understanding the dialogue content and accurately matching it with images. previous studies @xcite typically use dual-stream architectures for processing texts and images separately, followed by feature-based retrieval. recent improvements in pre-trained visual-language models @xcite @xcite have enhanced the accuracy of these systems by finetuning them for specific tasks. however, current research @xcite often overemphasizes inter-modal alignment and overlooks the complexities of dialogue context. dialogue context can be lengthy and cluttered with redundant information, such as greetings, small talk, and repeated phrases, thereby adding extra complexity to downstream retrieval tasks. long texts make it more difficult for the model to understand, and may also be constrained by the encoder's limitation on the length of the input tokens. consequently, it is crucial to better understand and extract image-related information from the dialogue context. in addition, dialogue context involves multi-turn exchanges, which may not directly describe the details of the images, leading to a significant disparity between the representations of dialogue contexts and images, directly affecting the matching results. to address the aforementioned problems, we propose a systematic framework (vcu) for balancing visual context understanding in dialogue for image retrieval. initially, to better understand the conversational content and mitigate interference from redundant information, we devise a two-step conversational content extraction process based on the large language model (llm). specifically, in the first step, leveraging the comprehension and generative capabilities of llm, we directly extract potential keywords related to images from the dialogue context. then these keywords serve as hints for generating sentence-form visual description in the second step. furthermore, to bridge the gap between the dialogue context and visual content, we enhance the visual representation using object-list texts associated with images. by computing similarity scores between token-level object-list text and patch-level image embeddings, we derive importance weights for each token, ultimately merging these weights with the original word embeddings and the global image embeddings to obtain enhanced image representation. lastly, inspired by @xcite , we introduce a method to construct auxiliary query text collections to balance the matching, thereby reducing the bias during similarity calculation and improving retrieval accuracy. to sum up, our contributions are threefold: ## preliminary task formulation for the task of dialogue-toimage retrieval, the model aims to select the image that best matches the current image-sharing intention from a pre-compiled database based on the dialogue history. we regard all the utterances preceding the image-sharing turn as dialogue history and denote it as @xmath0, u 2 , • • • , u m-1 , u m }, where u i represents the i-th turn of dialogue in the form of {user : content}, and m is the length of the dialogue history. at turn m+1, the model is required to retrieve an appropriate image from the pre-compiled image database @xmath1, o j } n @xmath2, where v j represents a candidate image and o j is the list of objects present in the corresponding image. throughout the inference process, given the dialogue history h, the model will retrieve the image v from image database d for sharing. pre-trained encoders clip @xcite is a multi-modal model that efficiently processes textual and visual data. pre-trained using a contrastive learning framework on a large dataset of text-image pairs, clip maps textual and visual embeddings into a common feature space, capturing semantic relationships between texts and images. in our research, we employ the text encoder ψ t and image encoder ψ v pre-trained by clip to extract embeddings of dialogue contexts and images. we then assess the alignment between these embeddings by calculating their cosine similarity. our optimization objective is to enhance the model's performance in cross-modal retrieval task, specifically dialogue-to-image retrieval, by maximizing the similarity of correctly matched text-image pairs while minimizing the similarity of mismatched pairs. ## methodology to address the challenges of conversational understanding and cross-modal matching in dialogueto-image retrieval task, we propose a systematic framework vcu, as illustrated in figure 2 . the framework mainly consists of three main components: first, employing llms for conversational extraction in section 4.1; second, using object-list texts to enhance visual embeddings in section 4.2; and finally, balancing matching by building an auxiliary query text collection in section 4.3. furthermore, we detail our learning objectives in section 4.4. ## conclusion in this work, we proposed the vcu framework to enhance the understanding of conversational context and improve cross-modal matching in dialogueto-image retrieval task. based on the dialogue history, our approach leverages the llms for two-step extraction, enhances visual representation through object-list texts, and constructs auxiliary query collections to balance the matching. these effectively bridge the gap between dialogue context and visual content. experimental results demonstrate that vcu outperforms baseline methods in imagesharing accuracy, highlighting the importance of considering the unique characteristics of dialogue. future research could further explore better conversational extraction strategies and the more effective utilization of image-side information.