Stark: Social Long-Term Multi-Modal Conversation with Persona Commonsense Knowledge (2024)

Abstract: Humans share a wide variety of images related to their personal experiences within conversations via instant messaging tools. However, existing works focus on (1) image-sharing behavior in singular sessions, leading to limited long-term social interaction, and (2) a lack of personalized image-sharing behavior. In this work, we introduce , a large-scale long-term multi-modal dialogue dataset that covers a wide range of social personas in a multi-modality format, time intervals, and images. To construct automatically, we propose a novel multi-modal contextualization framework, , that generates long-term multi-modal dialogue distilled from ChatGPT and our proposed image aligner. Using our , we train a multi-modal conversation model, 7B, which demonstrates impressive visual imagination ability. Furthermore, we demonstrate the effectiveness of our dataset in human evaluation. The code, dataset, and model will be publicly released after publication.

## introduction for decades, the development of empowering human-computer interaction has been steadily advancing across various domains (e.g., social dialogue @xcite , writing @xcite ), multifaceted ingredients (e.g., affective user's state @xcite , multi-perspective @xcite , multiple social skills @xcite ) and multimodality @xcite with the goal of increasing human satisfaction and engagement. to strengthen the interaction in a practicable real scenario, recent system @xcite have adopted the image-sharing behavior @xcite , an interaction frequently occurring via instant messaging tools, interpreting it as a communicative practice. consequently, previous studies have proposed multi-modal dialogue datasets through various methods, including crowdsourcing @xcite , social media @xcite , and distillation from large language models (llms) @xcite @xcite . however, existing datasets are limited in their representation of personalized image-sharing behavior over extended periods beyond a singular time (e.g., a few hours, days, weeks), preventing trained multi-modal dialogue models from seamlessly communicating with users in real-world human-bot interactive scenarios. for example, as shown in figure 1 , depending on who is the user (i.e., human's appearance), there is a user's appearance and user's personal experience inside the shared image. nevertheless, existing datasets regarding multi-modal dialogue do not consider multi-modality persona information (in to address this issue, we first introduce a largescale social long-term multi-modal conversation dataset with persona commonsense knowledge, stark , covering a wide variety of social personal dynamics (i.e., demographics, personal experience), more realistic time intervals, and personalized images. to construct stark, we propose a novel framework, mcu, that distills long-term multi-modal dialogue from a large language model (llm) foot_1 and our proposed plan-and-execute image aligner, powered by a personalized text-to-image generative model, image database retrieval, and web search, as shown in figure 1 . as a result of being grounded on various personal dynamics over a long period, stark contains more personalized multi-modal conversation dataset. in addition, even though stark is automatically constructed, stark show higher preferred quality compared to 2024.02.02: <@xmath0. with our stark dataset, we build a multimodal conversation model, ultron 7b, which is fine-tuned model on top of recent multi-modal language model @xcite . as a result, ultron achieves significant performance on dialogue-to-image retrieval task which implies the effectiveness of our dataset. in summary, our main contributions are as follows: 1) we propose the first large-scale social long-term multi-modal conversation dataset, stark , covering the personalized image-sharing behavior. 2) to construct stark, we propose a multi-modal contextualization framework, mcu, that generate a multi-modal dialogue over a time period by only providing basic demographic information. 3) using our dataset, we build a multimodal converstation model, ultron 7b. 4) through extensive experiments, we demonstrate the effectiveness and reliability of our dataset and framework in human evaluation and dialogue-toimage retrieval tasks. ## mcu: a multi-modal contextualization framework for conversation distillation inspired from recent study @xcite , we propose a mcu, a multi-modal contextualization framework for distilling long-term multi-modal dialogue from combination of large language model (llm) and our proposed plan-and-execute image aligner. specifically, mcu consists of eight steps: ## analysis of stark in this section, we conduct comprehensive analysis of stark in terms of diverse perspectives: comparison analysis to existing datasets ( § 4.1), multifaceted analysis ( § 4.2), and human evaluation ( § 4.3). ## ultron with stark, we train a multi-modal conversation model named ultron 7b. this model is designed to understand diverse social and personal dynamics along with previous interactions, enabling it to identify the appropriate moments for image sharing and retrieve relevant images based on the dialogue context. the overall architecture of ultron is illustrated in figure 6 . ## conclusion in this work, we first propose a social long-term multi-modal conversation dataset, stark , which is fully automatically constructed through our proposed framework, mcu. this framework comprises chatgpt and our proposed plan-and-execute image aligner. through extensive experiments, we demonstrate that our dataset has comparable quality to other existing datasets. additionally, using our dataset, we build a multi-modal conversation model, ultron, which achieves significant performance in the dialogue-to-image retrieval task.