NICE : Neural Image Commenting with Empathy (2021)

Abstract: Emotion and empathy are examples of human qualities lacking in many human-machine interactions. The goal of our work is to generate engaging dialogue grounded in a user-shared image with increased emotion and empathy while minimizing socially inappropriate or offensive outputs. We release the Neural Image Commenting with Empathy (NICE) dataset consisting of almost two million images and the corresponding human-generated comments, a set of human annotations, and baseline performance on a range of models. In-stead of relying on manually labeled emotions, we also use automatically generated linguistic representations as a source of weakly supervised labels. Based on these annotations, we define two different tasks for the NICE dataset. Then, we propose a novel pre-training model - Modeling Affect Generation for Image Comments (MAGIC) - which aims to generate comments for images, conditioned on linguistic representations that capture style and affect, and to help generate more empathetic, emotional, engaging and socially appropriate comments. Using this model we achieve state-of-the-art performance on one of our NICE tasks. The experiments show that the approach can generate more human-like and engaging image comments.

## introduction recent progress in the field of natural language processing (nlp) and computer vision (cv) has led to considerable advances in the domains of image captioning, visual question answering, visual dialog and visual storytelling @xcite @xcite @xcite @xcite @xcite . most image captioning tasks focus on generating literal descriptions of content either directly or in the form of searching or understanding. figure 1 : we present a dataset-nice and a novel pretraining model-magic for generating comments for user shared images. there are two examples for two nice dataset settings. in nice-setting i: in contrast to traditional image-captioning and image-grounded dialogue tasks, we focus on synthesizing content that is empathetic, emotional and engaging. nice-setting ii: the second setting aims to generate dialogue-style comments based on a comment topic and affect features. despite remarkable progress, developing intelligent dialogue agents that are capable of engaging in socially appropriate and empathetic conversations with humans is still very challenging. fig. 1 shows the examples of two images with comment threads for two nice-settings. the caption for the first image generated by a captioning model of the nice-setting i is "some houses are at the foot of a mountain". while this somewhat faithfully describes the image, imagine you posted the picture on social media and someone responded with that statement, would that spark an engaging conversation or feel like an empathetic response? probably not. a conversation is grounded not only in visible objects (e.g., houses and mountains) but also in events, actions and emotions (e.g., amazement at the grandeur of the mountain or a desire to climb it). emotions are important in meaningful conversations and especially in forming emotional connections. generating emotional comments would imitate human-like behavior, which is essential to human-machine interaction and conversation. in this work, we present the neural image commenting with empathy (nice) dataset with two task settings, and design a dialogue system that is capable of commenting on images in an emotional and engaging manner. to create a holistic measure of the performance for image commenting systems, we selected five dimensions that capture different conversational qualities: empathy, emotion, engagement, social appropriateness and relevance to the image-commenting pairs. we make the assumption that it is desirable for automatically generated dialogue to score well across all of these measures. emotion and empathy in comments are specified. emotion here is defined as the use of language that refers to, or reflects, affect and is a response to a specific stimulus (in this case the image and/or other comments). this is differentiated from mood which is affect not related to a specific stimulus but capturing a longer lasting feeling that might influence a whole conversation. empathy is defined as the ability to understand and share the feelings of another. we believe that this task will benefit various research fields such as vision-language and human-machine interaction. to summarize, the core contributions of this paper are: 1) we collect and release a large dataset 1 , nice, which contains almost two million images and more than seven million groups of comment dialogue conversation. 2) we define two different tasks on the two nice dataset settings including a sizable manually and automatically annotated portion. 3) we provide a benchmark results using established metrics (e.g., bleu, cider) and via human judgements of empathy, emotionality, engagement, social appropriateness and relevance. 4) we also introduce a novel pre-training approach, magic, to simulate human commenting on nice dataset, which aims to generate targeted comments on a given image weakly supervised by affect fea-1 users can access code and to download the dataset at our official website: https://nicedataset.github. io/ . use of the code and dataset are governed by an end user license agreement (eula) to avoid any potential violation of rights or terms of service. ## related work with the recent advances in deep learning, a growing number of researchers are interested in studying vision and language jointly. vision-language understanding has become one of the key components of conversational agents, such as cortana @xcite . a great deal of focus has been paid to image captioning @xcite @xcite , which typically focuses on literal descriptions of image content. however, in social conversations, people usually engage with others using language with emotions, opinions and subjectivity. for example, image commenting on human-machine interaction system has rich stylistic features. in this paper, we introduce the image comment generation task, where the aim is to build models that produce more engaging comments grounded in visual images. specifically, we present a pre-training model for this task. there are several pre-trained models that address various tasks across the language and vision space. large-scale pre-trained models have achieved stateof-art results on many natural language processing and generation tasks @xcite @xcite @xcite . pre-trained models learn representations using tasks such as predicting words based on their context. gpt-2 and ctrl are examples of language generation models that leverage pre-training. we use a well validated linguistic style representation to control the comment generation. we extract affect features for auto-labeling which are used to learn a control input related to word categories. some researches have also combined vision and language features in pre-trained models for various downstream vision-language tasks @xcite @xcite @xcite @xcite . one of the closest pre-trained generation models that compare with our work is the unified vision language pre-training (vlp) model. however, vlp focuses on generating image captions and lacks the ability to generate expressive, stylistic responses. to alleviate this problem, we propose our magic pre-training model to fill this gap and the proposed image commenting task offers a more natural setting for generating and evaluating comment dialogue with affection about visual content. ## conclusion and future work in this paper we present a new vision-language task called neural image commenting with empathy (nice) which extends image descriptions to comments with an emphasis on emotion and affect. we design contexts for this dataset based on different annotation schemes. for nice-setting ii, we propose a novel per-trained model, magic, for image commenting conditioned on affect and style features. we show that magic can help produce affective and emotional image comments. to facilitate research in this area, we release the nice dataset. the social language captured in this dataset has great value for training conversational systems. image commenting is an emerging area of research and ai systems for conversation are becoming increasiningly widely adopted. while we anticipate that the task we are proposing can have a significant positive impact in many domains (e.g., accessibility, storytelling, entertainment), we acknowledge that they can be abused (e.g., fake comment generation) and countermeasures may need to be developed. we hope that solving the nice task will benefit a wide range of applications including visual dialogue generation, visual question-answering and help create better social chat-bots and intelligent personal assistants.