Image-Chat: Engaging Grounded Conversations (2020)

Abstract: To achieve the long-term goal of machines being able to engage humans in conversation, our models should captivate the interest of their speaking partners. Communication grounded in images, whereby a dialogue is conducted based on a given photo, is a setup naturally appealing to humans (Hu et al., 2014). In this work we study large-scale architectures and datasets for this goal. We test a set of neural architectures using state-of-the-art image and text representations, considering various ways to fuse the components. To test such models, we collect a dataset of grounded human-human conversations, where speakers are asked to play roles given a provided emotional mood or style, as the use of such traits is also a key factor in engagingness (Guo et al., 2019). Our dataset, Image-Chat, consists of 202k dialogues over 202k images using 215 possible style traits. Automatic metrics and human evaluations of engagingness show the efficacy of our approach; in particular, we obtain state-of-the-art performance on the existing IGC task, and our best performing model is almost on par with humans on the Image-Chat test set (preferred 47.7% of the time).

## introduction a key way for machines to exhibit intelligence is for them to be able to perceive the world around them -and to be able to communicate with humans in natural language about that world. to speak naturally with humans it is necessary to understand the natural things that humans say about the world they live in, and to respond in kind. this involves understanding what they perceive, e.g. the images they see, what those images mean semantically for humans, and how mood and style shapes the language and conversations derived from these observations. in this work we take a step towards these goals by considering grounded dialogue involving open-ended discussion of a given image, a setting that is naturally fun for humans @xcite , and study neural conversational models for task. in particular, we explore both generative and retrieval models that handle multimodal dialogue by fusing transformer architectures @xcite for encoding dialogue history and responses and resnet architectures @xcite for encoding images. we propose ways to fuse those modalities together and perform a detailed study including both automatic evaluations, ablations and human evaluations of our models using crowdworkers. to train and evaluate such models, we collect a large set of human-human crowdworker conversations, with the aim of training a model to engage a human in a similar fashion, consisting of 202k diverse images and 401k utterances over the images, with 215 different style traits (e.g., optimistic, skeptical or frivolous) to promote engaging conversation. the dataset is made publicly available in parlai @xcite 1 . our results show that there is a significant gap between state-of-the-art retrieval and generative models on this task. our best fused retrieval models set a strong baseline, being preferred to human conversationalists 47.7% of the time. we show that both large-scale image and text pre-training, and utilization of style traits, are critical for best results. we then consider transfer to the existing image grounded conversations (igc) task of @xcite , where we obtain stateof-the-art results. ## related work the majority of work in dialogue is not grounded in perception, e.g. much recent work explores sequence-to-sequence models or retrieval models for goal-directed @xcite or chit-chat tasks @xcite . while these tasks are text-based only, many of the techniques developed can likely be transferred for use in multimodal systems, for example using state-of-the-art transformer representations for text @xcite as a sub-component. in the area of language and vision, one of the most widely studied areas is image captioning, whereby a single utterance is output given an input image. this typically involves producing a factual, descriptive sentence describing the image, in contrast to producing a conversational utterance as in dialogue. popular datasets include coco @xcite and flickr30k @xcite . again, a variety of sequence-to-sequence @xcite @xcite and retrieval models @xcite @xcite have been applied. these tasks measure the ability of models to understand the content of an image, but not to carry out an engaging conversation grounded in perception. some works have extended image captioning from being purely factual towards more engaging captions by incorporating style while still being single turn, e.g. @xcite @xcite @xcite . our work also applies a style component, but concentrates on image-grounded dialogue, rather than image captioning. visual question answering @xcite and visual dialogue @xcite are another set of tasks which employ vision and language. they require the machine to answer factual questions about the contents of the image, either in single turn or dialogue form. they do not attempt to model natural conversation, but rather assess whether the machine can perform basic perception over the image via a series of questions. there are some works which directly address dialogue grounded with vision. the work of @xcite assesses the ability to execute dialogue given video of computer soccer games. the work of @xcite investigates the use of sentiment-based visual features and facial expressions for emotional image-based dialogue. perhaps the most related work to ours is @xcite . their work considers (visual context, textual context, question, response) tuples, and builds validation and test sets based on 4k eventful images called image grounded conversations @xcite . no training data is provided, but instead the authors use twitter for that in their experiments. in contrast, we provide training, validation and testing sets over 202k images for our task (that do not overlap with igc), and consider a general set of images and dialogues, not just events and questions plus responses. in our experiments we also show strong transfer ability of our models to the igc task. while there are many ways to measure dialogue quality, human engagement is a popular metric. engagement itself can be measured in many ways @xcite but here we adopt the common approach of simply asking humans which speaker they find more engaging, following other works @xcite . ## image-chat the image-chat dataset is a large collection of (image, style trait for speaker a, style trait for speaker b, dialogue between a & b) tuples that we collected using crowd-workers, each dialogue consists of consecutive turns by speaker a and b. no particular constraints are placed on the kinds of utterance, only that we ask the speakers to both use the provided style trait, and to respond to the given image and dialogue history in an engaging way. the goal is not just to build a diagnostic dataset but a basis for training models that humans actually want to engage with. style traits a number of works have shown that style traits for image captioning help provide creative captions @xcite @xcite . we apply that same principle to image grounded dialogue, considering a set of 215 possible style traits, using an existing set from @xcite . the traits are categorized into three classes: positive (e.g., sweet, happy, eloquent, humble, witty), neutral (e.g., old-fashioned, skeptical, solemn, questioning) and negative (e.g., anxious, childish, critical, fickle, frivolous). we apply these to both speakers a and b, who will be assigned different style traits for each given conversation. images the images used in our task are randomly selected from the yfcc100m dataset foot_1 @xcite . dialogue for each image, we pick at random two style traits, one for speaker a and one for speaker a: peaceful b: absentminded a: fearful b: miserable a: erratic b: skeptical a: i'm so thankful for this delicious food. a: i just heard something out there and i have no idea what it was. a: what is the difference between the forest and the trees? oh look, dry pavement. b: what is it called again? b: it was probably a wolf coming to eat us because you talk too much. b: i doubt that's even a forest, it looks like a line of trees. a: not sure but fried goodness. a: i would never go camping in the woods for this very reason. a: there's probably more lame pavement on the other side! figure 1 : some samples from the image-chat training set. for each sample we asked humans to engage in a conversation about the given image, where the two speakers, a and b, each have a given provided style. b, and collect the dialogue using crowdworkers who are asked to both assume those roles, and to be engaging to the other speaker while doing so. it was emphasized in the data collection instructions that the style trait describes a trait of the speaker, not properties of the content of the image they are discussing. some examples from the training set are given in figure 1 . data quality during data collection crowdsourcers were manually monitored, checking to ensure they were following the instructions. poor performers were banned, with comments discarded. a verification process was also conducted on a subset of the data, where separate annotators were asked to choose whether the utterance fit the image, style, or both, and found that 92.8% of the time it clearly fit the image, and 83.1% the style, and 80.5% both. note, given that not all utterances should directly reference an image property or invoke the style, we do not expect 100%. overall dataset the overall dataset statistics are given in ## models we consider two major types of dialogue model: retrieval and generative. both approaches make use of the same components as building blocks. we use three sub-networks for the three modalities of input: (i) an image encoder, (ii) a dialogue history encoder; and (iii) a style encoder. in the retrieval model these are then fed into a combiner module for combining the three modalities. finally, there is a response encoder for considering candidate responses and this is scored against the combined input representations. an overview of the retrieval archictecture is shown in figure 2 . for the generative model, the three encoders are used as input, and a further decoder transformer is used for outputting a token sequence; beam search is applied. image encoder we build our models on top of pretrained image features, and compare the performance of two types of image encoders. the first is a residual network with 152 layers described in @xcite trained on imagenet @xcite to classify images among 1000 classes, which we refer to in the rest of the paper as resnet152 features. we used the implementation provided in the torchvision project @xcite . the second is a resnext 32 × 48d @xcite trained on 3.5 billion instagram pictures following the procedure described by @xcite , which we refer to in the rest of the paper as resnext-ig-3.5b. the representation r i of an image i is obtained by using the 2048-dimensional output of the image encoder as input to a feed-forward network: a multi-layer perceptron with relu activation units and a final layer of 500 dimensions in the retrieval case, and a linear layer in the generative case. style encoder to condition on a given style trait, we embed each trait to an n -dimensional vector to obtain its representation r s . we used @xmath0. dialogue encoder the entire dialogue history d is encoded into a fixed size vector r d using a transformer architecture @xcite , followed by a linear layer. such transformers have been shown to perform strongly on a variety of dia-logue tasks previously @xcite . we use a transformer with 4 layers, 300 hidden units, and 6 attention heads. the outputs are pooled (mean) to give a final vectorial encoding. we pretrain the entire encoder following the setup described in @xcite : we train two encoders on a next-utterance retrieval task on a reddit dataset of dialogues containing 1.7 billion pairs of utterances, where one encodes the context and another the candidates for the next utterance; their dot product indicates the degree of match, and they are trained with negative log-likelihood and k-negative sampling. we then initialize our system using the weights of the candidate encoder only, and then train on our task in either generative or retrieval mode. ## experiments we test our models on the image-chat and igc datasets using automatic metrics and human evaluations. we analyze the performance of the different module and architecture choices, as well as ablation studies to determine the importance of each of the model's inputs. ## conclusion this paper presents an approach for improving the way machines can generate grounded conversations that humans find engaging. focusing on the case of chit-chatting about a given image, a naturally useful application for end-users of social dialogue agents, this work shows that our best proposed model can generate grounded dialogues that humans prefer over dialogues with other fellow humans almost half of the time (47.7%). this result is made possible by the creation of a new dataset image-chat foot_2 . our work shows that we are close to having models that humans can relate to in chit-chat conversations, which could set new ground for social dialogue agents. however, our retrieval models outperformed their generative versions; closing that gap is an important challenge for the community. while our human evaluations were on short conversations, initial investigations indicate the model as is can extend to longer chats, see appendix g, which should be studied in future work. the next challenge will also be to combine this engagingness with other skills, such as world knowledge @xcite relation to personal interests @xcite , and task proficiency. g longer chat example a: transresnetret mm-sum (extraordinary) b: human (no style) a: it's the most beautiful mountain in the world! b: it looks really cool! i wonder if you can ski on it a: they are a mountain range i've climed many times! b: you have climbed that? that seems really hard a: i did it; i conquered this climb in only 7 hours and 10 minutes! b: wow, that's fast. i would ski down that but i would need a lift to take me up a: but could you? could you truly climb this? b: i really don't think i could a: climbing a mountain can give one a special strength, you need to experience it b: maybe one day on a smaller mountain a: it would take hard will and determination to scale that mighty peak figure 10: long-form conversation with the model. the model is given a style here, while the human is not.