Multimodal Persona Based Generation of Comic Dialogs (2023)

Abstract: We focus on the novel problem of persona based dialogue generation for comic strips. Dialogs in comic strips is a unique and unexplored area where every strip contains utterances from various characters with each one building upon the previous utterances and the associated visual scene. Previous works like DialoGPT, PersonaGPT and other dialog generation models encode two-party dialogues and do not account for the visual information. To the best of our knowledge we are the first to propose the paradigm of multimodal persona based dialogue generation. We contribute a novel dataset, ComSet, consisting of 54K strips, harvested from 13 popular comics available online. Further, we propose a multimodal persona-based architecture, MPDialog, to generate dialogues for the next panel in the strip which decreases the perplexity score by ~10 points over strong dialogue generation baseline models. We demonstrate that there is still ample opportunity for improvement, highlighting the importance of building stronger dialogue systems that are able to generate persona-consistent dialogues and understand the context through various modalities.

## introduction multimodal conversational agents build dialog systems that engage with modalities beyond text, in constructing next responses. they open up a novel direction of text-vision multimodality, where the agent is part of the scene, rather than being a distant observer. this facilitates research and creation of support based multimodal agents. these agents could be critical for various applications such as assistants for visually impaired, conversations with robots in physical settings, instruction following by a digital agent that is manipulating images, clarification discussions during a presentation and so on. such agents can help to promote literacy and language skills, as users engage with the generated dialogue to create their own stories. in all such cases, a natural conversational experience will be • i am son of odin, half-brother of thor. ## the comset dataset we contribute a novel comics dataset, comset, containing 13 popular english comic strips, obtained from gocomics. 1 each comic strip contains transcription and an image. we remove duplicate strips (re-broadcasts with minor modifications) based on levenshtein distance between transcripts. for each comic, we also obtained persona facts (representative personality traits) for each character by manually curating such information from websites like fandom, 3 wikipedia, foot_3 and tv tropes, 5 and paraphrasing all collected persona facts into first person english sentences. we describe data pre-processing and analysis in this section. ## methodology in this section we formalize the next utterance prediction task in the multi-modal persona-based dialogue setting for benchmarking comset and propose a novel baseline architecture mpdialog. ## conclusions and future work we propose a novel problem of next utterance prediction for comics given historical multimodal context consisting of previous utterances and panel images. we contribute a novel dataset, comset, which contains 53,903 strips, 159,610 panels and 238,484 utterances from 13 comics. we also propose a multimodal persona-based baseline model, mpdialog, which performs better compared to strong language-only and persona-based dialogue generation models, both in the seen comic and the unseen comic settings. we make our code and dataset publicly available 2 . in the future we plan to (1) focus on generation of humor-focused text, and (2) explore generation of next utterances and panel images together.