How do people talk about images? A study on open-domain conversations with images. (2022)

Abstract: This paper explores how humans conduct conversations with images by investigating an open-domain image conversation dataset, ImageChat. We examined the conversations with images from the perspectives of \textit{image relevancy} and \textit{image information} . We found that utterances/conversations are not always related to the given image, and conversation topics diverge within three turns about half of the time. Besides image objects, more comprehensive non-object image information is also indispensable. After inspecting the causes, we suggested that understanding the overall scenario of image and connecting objects based on their high-level attributes might be very helpful to generate more engaging open-domain conversations when an image is presented. We proposed enriching the image information with image caption and object tags based on our analysis. With our proposed \textit{image}^{+} features, we improved automatic metrics including BLEU and Bert Score, and increased the diversity and image-relevancy of generated responses to the strong baseline. The result verifies that our analysis provides valuable insights and could facilitate future research on open-domain conversations with images.

## introduction a picture is worth a thousand words. human communication often involves both text and images. understanding the image content and chatting about it is crucial for a chatbot to interact with people. current multimodal dialogue systems often equip with an object detector, and adapt similar architecture as text-based dialogue systems, except fusing text and image modalities through concatenation @xcite or an attention mechanism @xcite . to investigate whether an additional object detector is enough, and to understand what factors direct the conversation content when an image exists, we conducted a deep analysis of the imagechat dataset @xcite . we aimed to answer the following questions: (1) how is a conversation with image different from an open-domain conversation? is the image necessary or supplemental? how related is the image to the conversation topic? (2) does the topic of the three-turns conversation always be on the image? how does the transfer happen if the conversation topic transits from the image to others? can we predict the shift from the image? ( 3 ) what types of image information are used in the conversation? more specifically, we want to know how helpful image objects are in the conversation since baseline models usually use an object detection model as the image encoder. we addressed the questions by sampling and analyzing imagechat dataset from the aspects of image relevancy and image information, which are independent but intertwined. the former labels whether the given image is relevant to the conversation theme, and the latter marks the type of image information in utterances. the annotation results show that about 31% of utterances are not on the image-related theme, i.e., the utterances do not describe or could be generated without the image. in terms of the conversation, people transit conversation topics 54% of the time within three utterances, and surprisingly 7% of conversations entirely consist of non-image-related utterances. in these conversations where the image is optional, the topic often derived from attributes of one of image objects. we also discovered that 45% of utterances contain image objects, 23.7% have nonobject image information such as the description of events in the image, and 31.3% do not have any image information at all. this result hints that including comprehensive descriptions of the image beyond image objects could benefit the generation of utterances with image information. ## augmenting image information our analysis suggests the importance of the nonobject image information, which is often the scene in the image. therefore, we augmented the image feature by image caption to capture the scenario. we also found that explicitly using texts of objects tags and captions works better than fusing the latent vectors from captioning and object detection models. given object tags, we replace the single fullimage feature in the baseline model with several image region features to facilitate the extraction of image object information. ## conclusions in this paper, we analyzed the factors that influence open-domain conversations with images, from aspects of (a) image relevancy to the conversation theme and (b) image information in the conversation. according to our observations, open-domain conversations with images often branch off from one topic to another even within only three turns. the relation between the attributes of objects is the key to linking utterances with different themes. from the dynamics of image-relevancy, it is also interesting to notice that some conversation is a solo play where people just express themselves without responding. moreover, a comprehensive view of the whole image and the understanding of the image scene are also critical image information in utterances, besides image objects. therefore, we proposed incorporating image captions that could capture the overall image semantics beyond objects and may provide some hints to the links to other objects. we also found empirically that explicitly using texts of caption and object tags work better than incorporating captioning and object detection models in latent space. with our enhanced image features image + , we outperformed mmb on bleu, f1, and bert score, and generated more image-related and diverse conversation responses, confirming the effectiveness of our findings. we believe that our in-depth analysis and proposed findings would benefit the future research on the open-domain conversations with images.