Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models (2024)

Abstract: For dialogue systems, the utilization of multimodal dialogue responses, as opposed to relying solely on text-only responses, offers the capability to describe different concepts through various modalities. This enhances the effectiveness of communication and elevates the overall conversational experience. However, current methods for dialogue-to-image retrieval are constrained by the capabilities of the pre-trained vision language models (VLMs). They struggle to accurately extract key information from conversations and are unable to handle long-turn conversations. In this paper, we leverage the reasoning capabilities of large language models (LLMs) to predict the potential features that may be present in the images to be shared, based on the dialogue context. This approach allows us to obtain succinct and precise descriptors, thereby improving the performance of text-image retrieval. Experimental results shows that our method outperforms previous approaches significantly in terms of Recall@k.

## introduction in recent years, the landscape of online conversations has undergone a significant transformation thanks to the proliferation of instant messaging tools. unlike the past, when these exchanges were confined to text alone, today's conversations have evolved into a multimodal experience, incorporating elements like images and speech. the various communication modes not only enhance engagement but also prove invaluable for conveying complex information that can be challenging to communicate solely through text. @xcite highlighted the advantages of integrating images into conversations. for example, when discussing a topic with someone who may not grasp the concept, sharing an image can provide visual clarity for better comprehension. additionally, when precision is required to convey specific details about a subject, relevant images can be a more effective means of communication than text alone @xcite . consequently, the ability to generate responses using images is a crucial area of research in enhancing automatic dialogue systems. to equip these systems with the capacity to respond using images, a common method involves text-to-image retrieval, as demonstrated by previous work @xcite . in this approach, a model selects an appropriate image from a preconstructed image repository based on the context of the ongoing conversation. as storage costs decline and computational power advances, vision foundation models pretrained on large-scale, open-domain image-text pairs have emerged @xcite @xcite . these models have demonstrated outstanding performance in text-toimage retrieval tasks, excelling in both zero-shot and fully-trained scenarios. however, despite their impressive capabilities, these pre-trained visionlanguage models (vlms) still come with some limitations. one significant limitation is their suboptimal design for handling complete dialogue contexts effectively. often, they suffer from extracting key information comprehensively from the entire conversation. this constraint can lead to the loss of crucial contextual information, potentially undermining the model's overall performance. inspired by @xcite , we leverage the reasoning capabilities of large language models (llms) to generate the visual descriptor for the dialogue context. these descriptors encapsulate speculations about the image that the speaker intends to share, aiming to provide concise and precise cues for better text-to-image retrieval. our objective is to address the aforementioned limitations and enhance task performance. given that most vision models excel at identifying objects, scenes, and other visual elements in images @xcite , we employ a set of visually-focused queries, such as main subject and background scene, to bridge the gap between the ongoing dialogue and the pool of potential image candidates. these queries serve as templates for the llm to predict corresponding visual cues based on the dialogue context. we then utilize these queries and their resulting answers as dialogue-associated visual descriptors, as illustrated in the bottom part of our contributions can be summarized as 3-fold: our experiments validate the effectiveness of these designed queries. ## methodology our objective is to select an image from a preconstructed photo set {(v j , o j )} m @xmath0, where v j represents an image candidate and o j lists the objects appearing in v j . note that the object lists can be obtained through object detection in the pre-processing stage, and we treat this object information as given data. figure 1 illustrates the proposed framework, which introduces an innovative approach to estimate retrieval scores for each image candidate within a dialogue context. these scores are based on two criteria: scene-aligned and vision-aligned scores, both relying on visual descriptors. the scene-aligned score assesses whether the speculated visual cues align with the image-associated objects in a textual format. in contrast, the visionaligned score evaluates the alignment between the visual description and the image using visionlanguage models. ## experiments for the llm in (1), we utilized well-established llms with instruction tuning and reinforcement learning from human feedback (rlhf), including llama-2-7b-chat and llama-2-13b-chat @xcite . we employed greedy decoding for generating descriptors to ensure the correct format and reasoning capability. our pre-trained vision-language model (vlm) backbone is clip vit-b/32, and vlm training is executed on a single nvidia geforce rtx 2080 ti gpu with a batch size of 56. we utilize the adam optimizer with an initial learning rate of 1e-5. the weighting parameter λ was set to 1 to strike a balance between scene-alignment and vision-alignment. ## conclusion this paper presents a novel approach to enable multimodal dialogue systems for seamless photo sharing. by harnessing the reasoning capabilities of llms, our method generates precise visual cues from ongoing dialogue contexts. addressing challenges encountered in prior methods, such as accurately understanding extensive dialogue contexts and handling input length constraints, our approach exhibits clear superiority in experimental results. additionally, our comprehensive ablation study confirms the effectiveness of text-only vi-sual descriptors, indicating a promising pathway for bridging intricate dialogues and images through deep dialogue understanding via llms. this work not only advances photo sharing within dialogues but also lays groundwork for future sophisticated multimodal dialogue systems.