An animated picture says at least a thousand words: Selecting Gif-based Replies in Multimodal Dialog (2021)

Abstract: Online conversations include more than just text. Increasingly, image-based responses such as memes and animated gifs serve as culturally recognized and often humorous responses in conversation. However, while NLP has broadened to multimodal models, conversational dialog systems have largely focused only on generating text replies. Here, we introduce a new dataset of 1.56M text-gif conversation turns and introduce a new multimodal conversational model Pepe the King Prawn for selecting gif-based replies. We demonstrate that our model produces relevant and high-quality gif responses and, in a large randomized control trial of multiple models replying to real users, we show that our model replies with gifs that are significantly better received by the community.

## introduction conversations are central to many online social platforms. while most conversations are textbased, computer mediated dialog also affords alternative forms of communication, such as emoji or stickers like bitmoji, that allow users to express themselves @xcite . increasingly, these visual forms of communication have become common in social media @xcite , with a notable use of the reaction gif @xcite . these gifs are short video sequences that depict a particular scene and sometimes contain text that acts as a meta-commentary @xcite . as a result, conversations become multimodal where individuals reply to one another using combinations of text and gifs (figure 1 ). while conversational ai systems have been developed in a purely textbased setting, such systems do not capture the full multimodal behavior seen online. here, we study multimodal conversation by introducing new dialog models for selecting gif replies in conversation. pizzamagic: ahhhhh!!! the emnlp deadline is in 24 hours!! casualmodel: figure 1 : gif responses in conversation like the one shown above are embodied dialog that use visual imagery to convey reactions and emotions. this paper develops a system to select the appropriate gif response to messages. (pdf best viewed with adobe acrobat) conversation analysis is central to nlp and multiple approaches have analyzed this dialog structure @xcite @xcite and developed conversational agents to engage with people (e.g., @xcite @xcite . recent work has focused on generating open domain social chatbots that engage in sustained conversations in a natural way @xcite . because many of these systems are designed to support voicebased dialog, they overlook non-textual forms of interaction used in social media conversations. in parallel, multimodal nlp systems have been developed for image data, often focusing on image-totext tasks such as image captioning @xcite and visual question answering @xcite @xcite . more recent work has focused on the reverse text-to-image dimension, such as generating an image from a description @xcite . our work unites these two strands of research by integrating imagebased communication into conversational agents. our paper offers three main contributions. first, we propose the new task of selecting gif responses in multimodal conversation analysis and introduce a new dataset of 1,562,701 real-world conversation turns with gif replies. second, we introduce a new model pepe the king prawn that fuses image and text-based features to select a relevant gif response. in in-house experiments, we show that our model substantially outperforms strong baseline models at selecting the exact gif used in real data and, in a manual test of the quality of the best responses, achieves an ndcg of 0.8145 on the annotated test set. third, in a real-world test, we deploy our model as a part of a large-scale randomized controlled trial and show that the gif replies produced by our model are more highly voted by the community. data, code, and models are available at https://github.com/xingyaoww/gif-reply . ## gif communications gifs have been widely adopted in communication as a natural form of embodied speech where the visual imagery conveys emotions or a reaction as a response @xcite . these gifs commonly come from widelyknown cultural products, such as movies or television shows, which provides common knowledge for how they could be interpreted @xcite . however, a single gif may have multiple interpretations, depending on the context, cultural knowledge of its content, and the viewer @xcite . as a result, a single gif can serve multiple functions in communication @xcite . gifs have grown in their use through increasing affordances by platforms like tumblr, reddit, imgur, and twitter that allow gifs to be natively displayed like text in conversation threads @xcite . further, gif-based keyboards have been introduced that allow users to search for gifs that have been tagged with keywords or other metadata @xcite . yet, these technologies require that gif data be prepared with sufficient tags to be searchable or to have sufficient data to use collaborative filtering techniques for recommendations @xcite ). as a result, there is a clear gap in identifying appropriate response gifs directly from the text, which this work fills. ## data despite the widespread use of gifs, no standard dataset exists for text and gif replies. further, al-though platforms like twitter support gif replies, these gifs are not canonicalized to identify which responses correspond to the same gif. therefore, we construct a new dataset for this task by collecting responses, matching their images, and augmenting this data with metadata about the gif, where possible. a visual description of the whole procedure can be found in appendix figure 7 . ## gif reply models we introduce a series of models for producing a gif response in conversation. each model will select a gif from the 115k gifs in our dataset as a response to a text-based message. this task is related to but distinct from work on image-text matching @xcite , which aims to find an image describing a piece of text, or text-to-image (e.g., @xcite , which generates an image from a text description. here, we aim to select gifs that reflect natural continuations or reactions to a message in a dialog, akin to how gifs are used in social media. for all models, additional details on the training procedures and hyperparameters are provided in appendix a. the three models that follow use varying degrees of information about the gifs and text to select a response. ## evaluation we initially evaluate the methods in two ways. first, we use traditional classification-based evaluation, testing whether the models can reproduce the observed gif replies. however, some messages could have multiple valid gif responses. therefore, as a second test, we evaluate the model in a retrieval setting, measuring whether its most-probable responses are good quality for a message. experimental setup models are trained and tested on a dataset containing 1,562,701 tweet-oscar gif encoder tweet encoder tweet we 're getting married today ! bertweet-base transformer feature vector for downstream task 1 x 512 efficientnet gif encoder transformer encoder layer feature vector for downstream task 1 x 512 efficientnet efficientnet efficientnet efficientnet-b0 (shared weight) 4 selected frames 4 x 224 x 224 (after reshape) ocr 4 selected frames 4 x 224 x 224 (after reshape) oscar multimodal transformer extracted object names face woman [inter_frame_sep] face woman extracted caption aww , thank you feature vector for downstream task 1 x 512 extracted object feature vectors bottom-up attention gif pairs associated with 115,586 unique gifs, where 605,063 tweet-gif pairs are associated with at least one tag. using the finalized 241 unique tags as classes for multi-label classification, we split the dataset by stratify on tags using the iterative train-test split method provided by scikit-multilearn library @xcite to create a 80:10:10 train, dev, and test split which is finalized to train the models described in §4. following bertweet (nguyen et al., 2020), we preprocess tweets in our dataset using nltk tweettokenizer for tokenization, emoji package to translate emotion icons, and converted mentions and links to special "@user" and "httpurl" tokens. annotated data to test whether each model's predictions are valid responses, we annotate the ten most-probable gif predictions for a subset of the tweets in our test data. many tweets in our test set require substantial context to understand due to having few tokens, linking to urls that provide extra knowledge, mentioning other users in directed communication. these factors suggest social context or general knowledge aids in the recipient's understanding of the gif's intentions. while the model can still benefit from training on such examples, judging the appropriateness of response is difficult without access to the social context. therefore, to reduce interpretation ambiguity, we annotate only tweets without urls or user mentions and having at least 10 tokens. this process selects tweets with sufficient content to judge appropriateness independent of the larger social context. two annotators (the authors) were shown a list of potential gif responses for a tweet and asked to judge whether this is an appropriate gif response (a binary rating). gifs were selected from the ten most-probable replies for each system and collectively shown in random order to prevent knowing which system generated each reply. a total of 2,500 gif-tweet pairings were annotated. annotators attained a krippendorf's α of 0.462; while moderate agreement, this value is expected given known differences in how people interpret and value gif responses based on their familiarity with its content, message interpretation, and life-experience @xcite . we follow the evaluation setup from other retrieval-based dialog systems (e.g. @xcite and use normalized discounted cumulative gain (ndcg), which measures whether more appropriate gif responses are ranked higher. a gif's appropriateness score is the sum of annotators' ratings. ## field experiment to test the generalizability of our models and quality of their responses, we conduct a large-scale randomized controlled trial (rct) that has the models respond to real users and measure their perception of reply quality. foot_2 ## related work this work draws upon two strands of research from dialog systems and multimodal nlp. conversational dialog systems have traditionally been built upon large-scale dialog corpora from social media platforms @xcite such as twitter. our approaches are fundamentally information retrieval based systems that mirror the approach by text-based conversational systems that retrieve ex-isting messages from a large social media corpus as potential replies and rank these to select a response. our work mirrors models that use neural networks for ranking @xcite @xcite ; however, we note that many recent knowledge-grounded and open domain models use encoder-decoder methods to improve versatility and applicability (e.g., @xcite @xcite . generative approaches are likely inappropriate for gif-based conversation as gifs are more akin to mimetic artifacts that build on cultural knowledge @xcite , making synthesizing a new gif from scratch likely less effective. all three models used here rely on joint embedding spaces for gif and text. multiple works in nlp have been proposed to align these representations @xcite , often for particular applications such as visual question answering @xcite . recent work has focused on embeddings these media with a single encoder that takes both text and images as input (e.g., @xcite , in contrast to our model that uses separate image and text encoders (figure 3 ); these multimodal encoders are prohibitively computationally expensive to use in our setting during inference time, as the model would need to be run on each gif (and message) to rank replies, compared with our model that only needs to encode text. however, performance and efficiency improvements in aligning image and text representations would likely benefit our task. ## conclusion people like using gifs in online conversations-gifs are a fun and playful way to communicate. however, modern nlp conversational agents operate only by text. here, we introduce a new dataset of 1.56m conversation turns using gifs, including captions and metadata, and develop a new conversational model pepe the king prawn that selects appropriate gif responses for messages through comparing encoded gif and text representations. in two evaluations, we show that pepe is able to generate highly-relevant gif responses and in a largescale rct, we show that the gif replies from the pepe model received significantly higher scores from the general public. our work demonstrates the opportunity for using nlp methods to successfully engage in multimodal conversations.