STICKERCONV : Generating Multimodal Empathetic Responses from Scratch (2024)

Abstract: Stickers, while widely recognized for enhancing empathetic communication in online interactions, remain underexplored in current empathetic dialogue research, notably due to the challenge of a lack of comprehensive datasets. In this paper, we introduce the Agent for STICKERCONV (Agent4SC), which uses collaborative agent interactions to realistically simulate human behavior with sticker usage, thereby enhancing multimodal empathetic communication. Building on this foundation, we develop a multimodal empathetic dialogue dataset, STICKERCONV, comprising 12.9K dialogue sessions, 5.8K unique stickers, and 2K diverse conversational scenarios. This dataset serves as a benchmark for multimodal empathetic generation. To advance further, we propose PErceive and Generate Stickers (PEGS), a multimodal empathetic response generation framework, complemented by a comprehensive set of empathy evaluation metrics based on LLM. Our experiments demonstrate PEGS’s effectiveness in generating contextually relevant and emotionally resonant multimodal empathetic responses, contributing to the advancement of more nuanced and engaging empathetic dialogue systems.

## introduction increasing research indicates that utilizing stickers in online chats can effectively alleviate stress, augment personal happiness, and notably boost empathy @xcite @xcite . prior studies on stickers primarily concentrated on sentiment analysis @xcite and recommendation systems @xcite , overlooking their vast potential in empathetic response generation. most empathetic response generation tasks focus solely on textual modality @xcite @xcite , yet stickers 1 https://github.com/zhangyiqun018/stickerconv wow, what a fantastic surprise party! the room is just buzzing with happiness and laughter. it's so wonderful to see our dear friend so surprised and delighted. this moment is truly magical. this party is absolutely amazing, isn't it? the decorations, the music, and the energy in the room are all so contagious. it's like a burst of sunshine and love in here! the party sounds absolutely amazing! the decorations, music, and energy must be creating such a contagious atmosphere. it's like a burst of sunshine and love in the room! 😄🌞❤ i couldn't agree more! the party is like a kaleidoscope of colors and laughter, swirling together to create this incredible atmosphere of pure joy. it's like we're all caught in a whirlwind of happiness, and i don't want it to end! 😄🌈✨ it seems like you're really enjoying the party! the way you describe it as a kaleidoscope of colors and laughter, swirling together to create an atmosphere of pure joy, is so vivid and beautiful. it's like being caught in a whirlwind of happiness. i can understand why you wouldn't want it to end! 😄🌈✨ i'm so glad to hear that the surprise party turned out to be a fantastic one! the atmosphere must be filled with joy and laughter. it's heartwarming to witness the happiness of our dear friend. the moment sounds truly magical. 😊🎉 figure 1 : an example of multimodal conversation in the stickerconv. both parties can utilize the stickers to express their emotions, which enhances interactivity and expression. the assistant can empathize with the user according to the conversation (green text). in chats convey more abundant and intuitive emotional information, enhancing the expressiveness and emotional depth of responses. integrating stickers with textual communication and interspersing stickers within the dialogue can yield more varied and superior-quality empathetic replies. a primary challenge in integrating stickers into empathetic response generation is developing a high-quality dataset to support this innovative multimodal communication. to address this, we leverage large language model (llms) for dataset construction. llms, with their extensive world knowledge and text processing capabilities @xcite , demonstrate near-human annotation abilities @xcite . however, applying llms directly have limitations in empathetic tasks, excelling in responding to explicit human instructions but lacking proactivity, a critical aspect of empathy @xcite . em-pathy necessitates understanding others' emotions and the ability to actively express support and understanding @xcite . to mitigate this, we introduce a multi-agent system based on llms, agent for stickerconv (agent4sc). this system, through inter-agent interactions, utilizes stickers to simulate human-like dialogue scenarios. it not only generates text responses but also strategically selects suitable stickers, thereby effectively enhancing empathy. based on agent4sc, we build a multimodal empathetic dataset, stickerconv, that comprises 12.9k dialogue sessions and 5.8k unique stickers. stickerconv boasts an average of 5.22 stickers per dialogue session, mirroring the sticker usage patterns observed in human communication. figure 1 depicts an example of conversations in our dataset. to the best of our knowledge, this is the first multimodal empathetic dialogue dataset, with the particular utility of sticker as non-textual modal information to better facilitate empathy. although agent4sc effectively generates multimodal empathetic responses, it is limited by expensive inference costs and specific sticker databases. to further advance the research on multimodal empathetic dialogue, we develop an end-to-end multimodal empathetic response generation framework, pegs, with the ability to perceive and generate stickers. beyond the general ability to generate textual empathetic responses, pegs receives multimodal inputs and autonomously generates stickers based on the emotional and contextual aspects of the dialogue at the appropriate moment. furthermore, to simulate human communications on social media in the real world, our model supports interleaved multiple image and text inputs. misalignments between the modal quantities of predicted and golden responses can distort evaluation outcomes, and empathy is difficult to quantify due to its subjective nature. to address this, we propose a novel method for evaluating multimodal empathetic responses, focusing on empathy, consistency, and ranking. utilizing the extensive world knowledge and anthropomorphic abilities of llms, this approach provides solid support for assessing multimodal empathetic replies. in conclusion, the main contributions of this work are as follows: ## agent for stickerconv confronted with the pivotal challenge of a lack of datasets for multimodal empathetic response tasks, we craft our own dataset utilizing large language models (llms). nevertheless, llms experience difficulties in grasping nuanced human emotions and initiating actions beyond explicit directives. these limitations render llms and large multimodal models (lmms) less proficient in the generation of multimodal empathetic responses. in response to these issues, we introduce agent for stickerconv (agent4sc), a multi-agent system predicated on llm, devised to mimic human conversational patterns. figure 2 presents the overview of agent4sc. by integrating multiple modules and the strategic use of stickers, agent4sc aims to gen-erate emotional and varied empathetic responses, thereby overcoming the inherent deficiencies of llms in empathetic engagements. ## pegs we devise a multimodal empathetic response generation framework, pegs, with the ability to perceive and generate stickers. figure 3 illustrates the architecture of our framework. with different strategies to generate images, we derive three models based on this framework: pegs-ret/gen/rag, which denote the retrieval, generation, and retrieval-augmented generation methods to provide images. technically, we utilize vit-g/14 from eva-clip @xcite , q-former from blip-2 @xcite , and a linear layer to encode images. vicuna @xcite , a widely used language model in llms, is employed for language modeling. stable diffusion (sd) @xcite is employed as the image decoder for sticker generation. ## experiments experiments are conducted on stickerconv. we perform response predictions for all turns of each dialogue and consider all previous turns as context. ## conclusion we explored the concept of multimodal empathetic response and created the stickerconv, the first dataset specifically designed for multimodal empathetic dialogue. we developed the agent for stickerconv, a sophisticated llmbased multi-agent system capable of simulating human-like interactions using stickers, thereby creating multimodal empathetic responses. building upon the stickerconv, we developed perceive and generate stickers (pegs), an advanced multimodal empathetic dialogue framework. this framework adeptly perceives and generates stickers, effectively enhancing the communication experience. furthermore, we established a comprehensive set of evaluation metrics for multimodal empathy tasks based on llm. we are confident this work will be a valuable asset in advancing research in the field of multimodal empathetic dialogue systems.