Selecting Stickers in Open-Domain Dialogue through Multitask Learning (2022)

Abstract: With the increasing popularity of online chatting, stickers are becoming important in our online communication. Selecting appropriate stickers in open-domain dialogue requires a comprehensive understanding of both dialogues and stickers, as well as the relationship between the two types of modalities. To tackle these challenges, we propose a multitask learning method comprised of three auxiliary tasks to enhance the understanding of dialogue history, emotion and semantic meaning of stickers. Extensive experiments conducted on a recent challenging dataset show that our model can better combine the multimodal information and achieve significantly higher accuracy over strong baselines. Ablation study further verifies the effectiveness of each auxiliary task. Our code is available at https://github.com/nonstopfor/Sticker-Selection .

## introduction with the development of mobile messaging apps (e.g., whatsapp and messenger), visual content is getting more and more frequently used in our daily conversation, such as emojis and stickers. compared with emojis, stickers are larger images consisting of drawing characters, symbolic icons, and text titles, and are hence more expressive and versatile @xcite . users send stickers along with text to show intimacy, express strong emotion, and experience the enjoyment of creativity @xcite . despite the importance of stickers in daily communication, selecting stickers in open-domain dialogue hasn't been widely explored. in this paper, we address the task of selecting an appropriate sticker from a candidate set for an open-domain multi-turn dialogue. this task is a typical setting for various applications, e.g., automatically recommending stickers in messaging apps and building * work done during internship at wechat ai. given a dialogue history, the model needs to add a sticker to the last textual message which is the most appropriate one among a collection of candidate stickers (the one marked in the red rectangle). the words below in red denote the emotion or meaning of each sticker. considering the challenges of this task and the shortcomings of previous work, we propose a novel multitask learning method to improve sticker selection in open-domain multi-turn dialogue. we design three auxiliary tasks: 1) masked context prediction, which uses multimodal context to predict masked tokens in the dialogue history, aiming to understand the dialogue in the presence of the sticker; 2) sticker emotion classification, which utilizes the sticker's contextualized representation to predict its emotion, aiming to improve the model's understanding of sticker emotion; 3) sticker semantic prediction, which explicitly instills semantic understanding of stickers by training the model to reconstruct a sticker's semantic label based on the multimodal inputs. moreover, all these tasks help improve our model's joint modeling capability, as both our model architecture and task design require multimodal inputs and deep interactions between them. we evaluate the performance of our method on a recently proposed and challenging dataset. extensive experiments show that our multitask method achieves state-of-the-art performance. there are two contributions of this paper: visual dialogue. visual dialogue is a task to answer questions about the factual content of the real-world image @xcite . in contrast, selecting appropriate stickers in open-domain dialogue requires understanding sentiment and semantic expression of usergenerated, artistic style images. ## conclusion in this paper, we address the challenging task of selecting appropriate stickers in open-domain multiturn dialogue. we propose a multitask learning method with three auxiliary tasks to enhance the understanding of dialogues and stickers. experiments show that our model outperforms strong baselines, confirming the effectiveness of our multitask learning method for sticker selection. although our experiments are conducted on a chinese dataset, our methods are expected to work for other languages. all bert-based models and mod-gpt use the image encoder in the clip baseline. we set the clip image encoder's learning rate to 5e-7 and the text encoder's learning rate to 9e-6. for bertbased models, we set the learning rate to 9e-6 and 3 https://www.sbert.net/docs/ pretrained_models.html#image-text-models fix the image encoder following @xcite . the maximum epoch is set to 10. for the total loss, α is set to 0.05, β is set to 0.2 and γ is set to 0.1. all the hyperparameters are selected based on the validation set. the maximum training time for one epoch is about 5 hours on one single v100 gpu.