MPCHAT : Towards Multimodal Persona-Grounded Conversation (2023)

Abstract: In order to build self-consistent personalized dialogue agents, previous research has mostly focused on textual persona that delivers personal facts or personalities. However, to fully describe the multi-faceted nature of persona, image modality can help better reveal the speaker’s personal characteristics and experiences in episodic memory (Rubin et al., 2003; Conway, 2009). In this work, we extend persona-based dialogue to the multimodal domain and make two main contributions. First, we present the first multimodal persona-based dialogue dataset named MPCHAT, which extends persona with both text and images to contain episodic memories. Second, we empirically show that incorporating multimodal persona, as measured by three proposed multimodal persona-grounded dialogue tasks (i.e., next response prediction, grounding persona prediction, and speaker identification), leads to statistically significant performance improvements across all tasks. Thus, our work highlights that multimodal persona is crucial for improving multimodal dialogue comprehension, and our MPCHAT serves as a high-quality resource for this research.

## introduction with the rapid advance of conversational ai systems in recent years, developing self-consistent dialogue agents has been studied much @xcite . considerable research aims to endow dialogue agents with persona, which represents an individual's personality @xcite . in particular, researchers have exploited textual description of persona, for example, in the form of unstructured sentences @xcite , structured key-value attributes (e.g., age, gender, location) @xcite and personality types (e.g., big-five) @xcite . therefore, dialogue agents with persona have been found to 42 comments reply u/userb • 2 weeks ago curious, what would you estimate the ratio of acceptable shots to unacceptable shots is? however, existing studies restrict the role of persona only to personal facts @xcite or personalities @xcite , while it should be explored in multi-faceted ways @xcite . more than factual information, episodic memory @xcite , which is the memory of everyday events or personal experiences connected to the self and autonoetic consciousness @xcite , should be included in persona component. @xcite further supports this assertion by arguing that episodic memory plays a significant role in shaping personal identity, which in turn can influence one's persona. since episodic memories are often represented in the form of visual images or history scenes @xcite , we propose to study the multimodal persona, which consists of a set of image-sentence pairs describing memorable moments as shown in figure 1 . furthermore, visual information can complement textual information, which often lacks an explicit description of appearance or measurable quantities @xcite . in this work, we contribute to the personabased dialogue research in two important ways. first, we introduce a new multimodally personalized dialogue dataset named multimodal persona chat (mpchat), where personas reveal speakers' episodic-memories using both text and images. to the best of our knowledge, mpchat is the first dataset that supports multimodal persona in dialogue. to collect episodic-memory-based multimodal personas, we source users' posts from social media reddit. we carefully design a pipeline to curate multimodal conversation data that are wellgrounded on multimodal personas 1 . second, based on mpchat, we propose three retrieval-based dialogue tasks as benchmarks for multimodal persona-grounded dialogue understanding: next response prediction, grounding persona prediction, and speaker identification. by incorporating our proposed multimodal persona, we observe statistically significant performance improvements across all tasks. consequently, our work illustrates the significance of multimodal persona in enhancing multimodal dialogue comprehension, and our mpchat provides a valuable resource for the research, given its well-grounded dialogues (especially responses) on multimodal personas. ## the mpchat dataset we collect a multimodal persona-grounded dialogue dataset named mpchat (multimodal persona chat). the objective of mpchat is to help a conversational agent utilize its episodicmemory-based persona, consisting of both linguistic and visual information, to produce personagrounded responses. to cover a wide range of episodic-memory-based multimodal persona, we source posts from social media reddit. however, dialogue with a multimodal persona introduces two new challenges. first, it is harder to collect persona image-sentence pairs than to collect personas sentences. second, it is also difficult to collect dialogue instances grounded on speakers' multimodal personas since each utterance should be grounded on not only persona sentences but also persona images, which may require more finegrained information with additional commonsense knowledge @xcite . to overcome these challenges, we design the process of data construction as follows. ## task definition as benchmark tasks for mpchat, we consider three retrieval tasks as follows. specifically, the dialogue dataset d is a list of n dialogues, each of which consist of (c, r, p ), where a context @xmath0, c t ) contains a context image c i and context text c t (i.e., context utterances), r is a response to context c, and a persona set @xmath1, p t 1 ), ..., (p i m , p t m )} is a set of @xmath2. we below describe each task setting. next response prediction. the goal of this task is to predict the next response r * based on pr(r|c, p, r c ), from a response candidate set r @xmath3, r 2 , ..., r cr }, as shown in figure 3 . the response candidate set r c contains a correct response r * and c r -1 randomly sampled test responses. grounding persona prediction. this task aims at predicting the persona element p * , which grounds r (i.e., labeled as entailed in § 3.3) based on pr(p|c, r, p , p c ) or pr(p|c, p , p c ). p @xmath4, p 2 , ..., p cp } is a persona (element) candidate set, which includes a correct persona element p * and c p -1 randomly sampled persona elements from other speakers. p is the speaker's remainder persona set, a set of m -1 persona image-sentence pairs in p except p * . note that we consider two cases of whether r is given or not. if r is not given (i.e., no-response case), then a model needs to retrieve the most likely persona element p * based on a given context c and a remainder persona set p before producing a response r. if r is given (i.e., response case), a model predicts p * that grounds r, which is much easier than the former case. speaker identification. finally, we predict the speaker (with his/her multimodal persona set) p * who spoke the response r based on pr(p |c, r, p c ), from a speaker candidate set p @xmath5, p 2 , ..., p c p }. the speaker candidate set p c includes a correct speaker p * and c p -1 randomly sampled speakers. following humeau et al. (2020); zhong et al. (2020); shuster et al. (2020); lee et al. ( 2021 ), we use recall@1 and mean reciprocal rank (mrr) as evaluation metrics, and set the number of retrieval candidates c r , c p , and c p to 100. ## models to solve the proposed retrieval-based dialogue tasks, we first define a set of unimodal encoders for the input of persona image and text (p i , p t ), context image and text (c i , c t ), and a response r. we then construct multimodal persona-aware models by combining these modules based on input components for each task. note that we design our models to be simple and standard, to investigate the characteristics of our dataset. text encoder. we use a transformer @xcite as the text encoder for context text c t , persona sentences p t , and a response r. we test two initialized weights of sbert 2 @xcite and the clip-vit-b/32 text model @xcite . for a persona input p t , we encode the concatenation of m persona sentences. the representation of each text input (h c t , h p t , h r ) is obtained by the mean-pooled output of the entire sequence for sbert or the hidden state of the first token [cls] (for clip), followed by a linear layer. image encoder. we encode a context image c i and a set of persona images p i using a single grid-based vit-b/32 @xcite and clip-vit-b/32 vision model @xcite 2 https://huggingface.co/sentence-transformers/ multi-qa-distilbert-cos-v1 . persona image encoder persona text encoder response encoder context text encoder context image encoder retrieval-based dialogue model p t 1 ... p t m c t r p i 1 ... p i m c i dot product b: jindo rescue gets confused for cayote all b: first c2c project, a little mat b: portrait mode on dog a: wow! a very fox b: jindo rescue! he is skinny as he has eating response score ... ... ## experiments the main goal of our experiments is to verify that multimodality from images and text indeed helps better understand persona-based dialogues, and our mpchat is properly collected for this purpose. thus, we design our experiments as follows. (1) our models are rather simple and standard, as discussed in §5. (2) we compare our models that take advantage of full inputs with several baselines that use only parts of them. ## conclusion we studied episodic-memory-based multimodal persona-grounded dialogue, and introduced mpchat as the first multimodal personagrounded multi-turn dialogue dataset. we proposed three retrieval-based dialogue tasks to evaluate the effectiveness of multimodal persona. with the help of multimodal persona, all of the proposed models exhibited better dialogue comprehension abilities. our empirical results showed that dialogues (especially responses) in mpchat are well grounded on multimodal personas as intended. one interesting future work would be to expand mpchat in both the size (e.g., scaling up the number of dialogues and personas) and the scope (e.g., adding audio/video modality).