P hoto C hat: A Human-Human Dialogue Dataset With Photo Sharing Behavior For Joint Image-Text Modeling (2021)

Abstract: We present a new human-human dialogue dataset - PhotoChat, the first dataset that casts light on the photo sharing behavior in online messaging. PhotoChat contains 12k dialogues, each of which is paired with a user photo that is shared during the conversation. Based on this dataset, we propose two tasks to facilitate research on image-text modeling: a photo-sharing intent prediction task that predicts whether one intends to share a photo in the next conversation turn, and a photo retrieval task that retrieves the most relevant photo according to the dialogue context. In addition, for both tasks, we provide baseline models using the state-of-the-art models and report their benchmark performances. The best image retrieval model achieves 10.4% recall@1 (out of 1000 candidates) and the best photo intent prediction model achieves 58.1% F1 score, indicating that the dataset presents interesting yet challenging real-world problems. We are releasing PhotoChat to facilitate future research work among the community.

## introduction as instant messaging tools gain enormous popularity in the recent decades, sharing photos as an approach to enhance the engagement of an online messaging conversation has become a pervasive routine communicative act @xcite . a survey conducted in 2010 reveals that 74% of teenagers in the us reported messaging a photo or video using their cell phone @xcite . in britain, almost 70% of the internet users shared photos in 2013 @xcite . considering the proliferation of photo sharing, it's desirable to have an intelligent system that can assist users efficiently engaging in this process, i.e. suggesting the most relevant photos in correct timings. in order to achieve this goal, the intelligent system is expected to not only understand how humans communicate with each other, e.g. the natural language human speak, but also perceive images as human do. how to facilitate building such multimodal system is the goal of this paper. though recently many image-text tasks have been proposed and are being actively studied to bridge language and vision, the majority of them are formulated as choosing or composing the text based on the understanding of given images, e.g. image captioning @xcite , visual question answering @xcite , visual commonsense reasoning @xcite , and image-grounded dialogue generation @xcite . contrary to these tasks, the photo sharing task focuses on the reverse process, i.e. selecting the image based on the understanding of text, as well as proposing different and unique challenges. firstly, different from the above popular multimodal tasks, in photo-sharing task, the dialogue doesn't often explicitly mention the main visible content in the image. instead of the main object of the photo, sometimes the background story, complemented by human imaginations, can be the focus of the chat. figure 1 shows such an example, in which the person who shares the photo describes the event location "court" and the occupation "attorney" instead of the main object "lady" in the image. secondly, the dialogue is not guaranteed to be relevant to the image. for instance, it often contains greetings and chit-chats of other topics, as the first two turns in figure 1 shows. in order to suggest the relevant photo, a smart system needs to decide which part of the dialogue can be used for suggesting the image. in contrast, in the traditional image-text tasks, the correct text is designed to be highly correlated with the image and has few distracting content. these photo sharing characteristics makes inferring the connection between the image and textual utterances challenging. to highlight these challenges, we create pho-tochat -a human-human dialogue dataset in which one photo is shared from one person to the other during the conversation foot_0 . it is, as far as we know, the first dataset that captures the photo sharing activities. we selected images from openimage v4 dataset @xcite as shared photos and used crowdsourcing plugins to generate 12,286 dialogues with an average of 10 turns per dialogue. during the dialogue collection, the photo is only visible to the side who is instructed to share the photo and then to both sides after it is being shared. based on the collected dataset, we propose two tasks that are essential for building a photo suggest system: photo-sharing intent prediction task that predicts whether one intends to share the photo in the next conversation turn, and dialogue-based image retrieval task that retrieves the most relevant photo given the dialogue context. for both, we build baseline models, report and analyze their performances. the best photo-sharing intent prediction baseline model achieves 58.1% f1 score with 58.2% precision and 57.9% recall. the best cross-attention image retrieval model achieves 10.4% recall@1 out of 1000 candidates. we also propose a dual-encoder model that leverages object labels to encode image features, which achieves the best performance among all the models w/o cross-attention mechanisms. in summary, our main contributions are: ## related work with the recent advances in deep learning, plenty of image-text datasets have been created and new image-text tasks are proposed based on them. these datasets have greatly stimulated the development of joint image-text models. in this section, we review the widely used image-text datasets and the state-of-the-art (sota) approaches for solving the image-text problems. ## dataset creation we select photos from open image dataset v4 (oid) @xcite ) and collect openended conversations on amazon mechanical turk. below describes the detailed image filtering, conversation generation, and data verification steps to ensure data quality. ## dataset statistics the collected dataset consists of 10,917 unique images and 12,286 dialogues. one image is shared in each dialogue. based on the object labels of the shared image, we classify the dialogues into four categories: people, food, animals, and daily products. we split the dialogues into 10,086 train, 1,000 dev, and 1,000 test sets while keeping roughly the same distribution of the category across the splits. the detailed statistics of each split and in total are shown in based on the statistics in the table, the average number of turns per dialogue is 12.7 and the average number of tokens per turn is 6.3. since two sides are not restricted to speak alternatively, if the consecutive turns from the same side are combined as one turn, which is the conventional setting of other dialogue datasets, the average number of turns per dialogue and the average number of tokens per turn become 9.5 and 8.5. on average, people converse for 7 turns before sharing the photo. ## task definition we decompose the problem of building a smart photo-suggest system into two separate tasks. the first is to detect if the user has the intent to share the photo in the next turn, which we call photo-sharing intent prediction task. the second is to retrieve the photo based on the dialogue context, which we call image retrieval task. below describes the formal formulation of the problem settings. let @xmath0, p 2 , ..., p m } be the photo set where each p @xmath1, l i ), @xmath2, m ] consists of image a i and a list of objects l i in it. given the dialogue @xmath3, ..., t h , p k , t h+1 , ..., t n } where two participants speak alternatively, t j (@xmath4, n ]) and p @xmath5 from left to right starting from the second column, the name of each column means "the unique number of images", "the number of dialogues", "the number of dialogues about people/food/animal/product", "the number of turns", "the number of turns when counting consecutive turns of the same speaker as one turn", and "the number of tokens". turns in which photos are shared are excluded in the calculation. we also define the speaker information @xmath6, s 2 , ..., s n } where s j (@xmath7, n ]), either 0 or 1, denotes the speaker of turn j. ## baselines 6.1 photo-sharing intent prediction model to establish the baselines, we fine-tune three sota pretrained models -bert @xcite , albert @xcite , and t5 (raffel et al., 2020), as the pretrained models have achieved remarkable performance in many nlp tasks. to adapt bert and albert to our settings, we concatenate all the previous turns (t 1:j in equation 1) by [sep] and prepend the concatenated text with @xcite to generate the input to the model. we use the speaker information s 1:j as the segment id of the input. the output of [cls] token is fed into two fully-connected layers, of which the output dimensions are respectively 128 and 2 to generate the final prediction. to utilize t5, we concatenate t 1:j by [sep] and prepend the text with "predict share intent:" as the model input. we use cross entropy loss for all three models. ## conclusion we collected a 12k high-quality dialogue dataset that contains photo sharing activity via crowdsourcing. to facilitate research on building intelligent photo-suggest system, we have introduced two new challenging tasks that aim at improving the photo-sharing experience: photo-sharing intent prediction task and image retrieval task. that is, when given a dialogue, the system should predict whether the user has the intention to share the photo and which photo is suitable to be shared. we built baseline models for both tasks and report their performance with detailed analysis. besides the proposed two new tasks, our dataset can potentially be used in other dialogue related tasks, such as dialogue generation in the multimodal dialogues, as well as inspiring new research topics, such as composing automatic reply to the photos sent from others. we hope our dataset and modeling work can be beneficial for studies that focus on the interplay between image and dialogue.