Large Language Models can Share Images, Too! (2024)

Abstract: This paper explores the image-sharing capability of Large Language Models (LLMs), such as GPT-4 and LLaMA 2, in a zero-shot setting. To facilitate a comprehensive evaluation of LLMs, we introduce the photochatplus dataset, which includes enriched annotations (ie intent, triggering sentence, image description, and salient information). Furthermore, we present the gradient-free and extensible Decide, Describe, and Retrieve () framework. With extensive experiments, we unlock the image-sharing capability of equipped with LLMs in zero-shot prompting, with ChatGPT achieving the best performance.Our findings also reveal the emergent image-sharing ability in LLMs under zero-shot conditions, validating the effectiveness of . We use this framework to demonstrate its practicality and effectiveness in two real-world scenarios: (1) human-bot interaction and (2) dataset augmentation. To the best of our knowledge, this is the first study to assess the image-sharing ability of various LLMs in a zero-shot setting. We make our source code and dataset publicly available at https://github.com/passing2961/DribeR.

## introduction people often share a variety of images during interactions via instant messaging tools. in practice theory, this is referred to as photo-sharing behavior @xcite , which is interpreted as a communicative practice. 2 this behavior operates internally through a two-stage system @xcite : (1) when to share and (2) what to share. for example, as shown in figure 1 , we first discern the appropriate moment (i.e., decision) for sharing an image with certain intent based on one sentence that invokes the image-sharing behavior. then, we 1 https://github.com/passing2961/driber 2 from now on, we refer to this as image-sharing behavior, given that "image" is a broader concept than "photo," thereby providing more flexibility to language models. share a relevant image at that moment, either by searching the internet or using photos taken on our mobile devices. this work primarily focuses on unlocking the image-sharing behavior capabilities of large language models (llms) in a zero-shot manner. however, recent studies related to multi-modal dialogue exhibit two significant limitations; (1) over-simplification. as shown in figure 1 , we humans decide to share images with an underlying intent, such as visual clarification, due to a certain triggering sentence (e.g., "what did you eat?") in the previous dialogue context. nevertheless, existing studies @xcite have reduced this complex behavior to a binary format (i.e., "yes" or "no"), which oversimplifies the complexity of image-sharing behavior. this sim-plification limits a comprehensive understanding of image-sharing behavior. (2) limited understanding of dialogue. since the key evidence is scattered across the entire dialogue context @xcite , understanding the linguistic cues underlying dialogue context is critical @xcite for retrieving images relevant to the dialogue. nevertheless, existing systems @xcite @xcite have primarily leveraged a dialogue-image matching score based on the cosine similarity between the two modalities (i.e., dialogue and image) to retrieve the relevant image at the image-sharing moment. this approach leads to a lack of capacity to comprehend the dialogue context due to the limitation of dual encoder structures (e.g., clip) in fully grasping the dialogue context @xcite . this work explores whether llms contain the image-sharing capability, primarily focusing on a zero-shot performance. to this end, we introduce decide, describe, and retrieve (driber), a gradient-free, extensible, and generalizable framework to unlock this image-sharing capability of llms through in-context zero-shot learning. broadly, driber consists of three stages: (1) deciding the image-sharing behavior with the intent, (2) describing the image description relevant to the previous dialogue, and (3) retrieving the relevant image to the image description. the overall pipeline is illustrated in figure 3 . in addition, we introduce photochat++, an extended version of photochat. photochat is a multi-modal dialogue dataset constructed via crowdsourcing for human-human interaction. photochat++ contains six intent labels, a triggering sentence, and salient information (e.g., "words" or "phrases") to invoke the image-sharing behavior. in our experiments, we successfully unlock the image-sharing capability of llms in a zero-shot setting with the aid of driber, with chatgpt achieving state-ofthe-art performance. using the photochat++ dataset, we demonstrate that image-sharing is a challenging task for both humans and llms even if we apply the few-shot setting and chain-of-thought reasoning. our extensive experiments further confirm that our framework is effective and versatile in real-world applications, specifically in (1) human-bot interaction dialogue and (2) dataset augmentation. in summary, our main contributions are as follows: 1) we introduce the decide, describe, retrieve (driber) framework, designed to evalu-ate the image-sharing ability of llms in a zeroshot setting. 2) for a comprehensive assessment of llms' image-sharing capabilities, we present the photochat dataset, enriched with additional information such as intent, triggering sentences, image descriptions, and salient information. 3) compared to the existing method, experimental results show that driber with llms achieves competitive zero-shot performance, even without additional training. 4) to the best of our knowledge, this is the first study to test the image-sharing capability of llms through zero-shot prompting. ## overview of photochat++ the photochat++ provides additional information (i.e., intent, triggering sentence, image description, and salient information) related to the imagesharing behavior by annotating the original pho-tochat test set. the purpose of this dataset is to thoroughly assess the image-sharing capability of llms based on the internal operation system of humans. we describe the details of the human annotation procedure in appendix e. ## evaluating image-sharing capability to verify the performance for each stage, we measure the performance on various evaluation metrics. each metrics are described as follows. ## related work multi-modal dialogue dataset. existing studies predominantly fall into two categories, depending on whether the image in the dialogue is grounded or sharing. image-grounded dialogue tasks are designed to answer questions @xcite @xcite or generate natural conversations @xcite @xcite about given images. nevertheless, it is common to share images pertinent to dialogue contexts in everyday conversations for the purpose of reinforcing social bonding, as well as enhancing engagement and interest. recent studies have proposed datasets that encapsulate this image-sharing behavior. this has been achieved by collecting a human-human dialogue dataset (photochat) via crowdsourcing @xcite , a large-scale dataset (mmdialog) from social media @xcite , or constructing datasets automatically using vision-and-language models @xcite . in this work, our focus is exclusively on the pho-tochat dataset to gain a deeper understanding of the image-sharing capabilities of llms. we do not include automatically constructed datasets or the mmdialog due to the considerable expense associated with conducting experiments using llms. prompting large language models. recent studies have witnessed the success of large language models, such as instruct gpt-3 @xcite , chatgpt (openai, 2023a), gpt-4 (openai, 2023b), in a zero-/few-shot performance. the use of these models, in conjunction with "prompt engineering," has unlocked the abilities of llms, even emergent ones @xcite , across various tasks. these tasks range from dialogues @xcite @xcite , complex reasoning tasks @xcite , and theory-ofmind @xcite , to image classification @xcite @xcite and multi-modality @xcite . ## conclusion in this paper, we explore the image-sharing capabilities of llms in a zero-shot prompting by introducing a three-stage framework: decide, describe, retrieve (driber). our extensive experiments demonstrate the effectiveness of our framework in enhancing zero-shot performance across both stages, with chatgpt achieving state-of-theart performance. we also reveal that the imagesharing ability is an emergent ability in the zeroshot prompting. with extensive experiments, we reveal the effectiveness of our framework on two useful applications. in future works, we will assess this capability in a few-shot setting using additional multi-modal dialogue datasets. we will construct a personalized multi-modal dialogue dataset using our framework.