JDDC 2.1: A Multimodal C hinese Dialogue Dataset with Joint Tasks of Query Rewriting, Response Generation, Discourse Parsing, and Summarization (2022)

Abstract: The popularity of multimodal dialogue has stimulated the need for a new generation of dialogue agents with multimodal interactivity.When users communicate with customer service, they may express their requirements by means of text, images, or even videos. Visual information usually acts as discriminators for product models, or indicators of product failures, which play an important role in the E-commerce scenario.On the other hand, detailed information provided by the images is limited, and typically, customer service systems cannot understand the intent of users without the input text.Thus, bridging the gap between the image and text is crucial for communicating with customers.In this paper, we construct JDDC 2.1, a large-scale multimodal multi-turn dialogue dataset collected from a mainstream Chinese E-commerce platform, containing about 246K dialogue sessions, 3M utterances, and 507K images, along with product knowledge bases and image category annotations. Over our dataset, we jointly define four tasks: the multimodal dialogue response generation task,the multimodal query rewriting task, the multimodal dialogue discourse parsing task, and the multimodal dialogue summarization task.JDDC 2.1 is the first corpus with annotations for all the above tasks over the same dialogue sessions, which facilitates the comprehensive research around the dialogue.In addition, we present several text-only and multimodal baselines and show the importance of visual information for these tasks. Our dataset and implements will be publicly available.

## introduction with the development of the internet, multimodal dialogue has become much more natural and prevalent in many scenarios, such as e-commerce, restaurant, travel, and so on, which stimulates research on dialogue agents with multimodal perceptions, understanding the interaction between vision and language. take the scene of e-commerce as an example, when users resort to customer service for solving the difficulties they encounter in the process of online shopping, there could be various forms of information, including text, images, or even videos, which establishes a great challenge for customer service systems to understand users' requirements. generally, users tend to express their needs with text. while sometimes, the text fails to convey enough information, and in that case, users may upload some images. for example, in figure 1 , images are used for distinguishing different product models for the same brand or used for identifying the location and cause of product failures. therefore, there is an urgent need for customer service systems to understand multimodal information sent by users to provide proper responses. multimodal information processing has been widely explored recently. researches on image captioning @xcite @xcite , visual question answering @xcite @xcite , multimodal machine translation @xcite @xcite , multimodal summarization @xcite @xcite @xcite , and visual or multimodal dialogue @xcite @xcite have made remarkable progress. however, most of the existing researches on multimodal dialogue are based on independent single-turn question answering or simulated dialogue flows, and the application of multimodal multi-turn dialogue in real e-commerce scenarios remains to be studied. in this paper, we build a large-scale multimodal dialogue dataset in e-commerce that aims to boost the research on multimodal dialogue. early work @xcite @xcite @xcite ., query 1: 我插上20分钟了，锅不热啊。(i have plugged it in for 20 minutes, the pot is not hot.) response 1: 调节大档了吗？灯亮吗？(have you adjusted the big gear? is the light on?) query 2: response 2: 插头没有插进去哦。(the plug is not inserted) query 1: 电热饭盒怎样使用啊？(how to use the electric lunch box?) 有操作视频吗？ (is there an operation video?) response 1: 这个没有的呀。(there is no such thing.) query 2: response 2: 夹层需要放上。(the mezzanine needs to be placed.) 蛋羹放这上面？ (put custard on this?) rewritten query: 蒸蛋羹的锅放在塑料笼屉上面吗? (is the steamed egg custard pot on top of the plastic cage?) multimodal dialogue response generation task multimodal query rewriting task query 1: 刚收到的，怎么有点脏，而且有点脱线？(i just received it. why is it a little dirty and unravelling? ) response 1: 确实非常抱歉哦，为您申请15元的现 金补偿，您看可以吗？。 (i'm really sorry. i'll apply for a cash compensation of 15 yuan for you. it that ok?) summary: 用户表示商品刚收到有点瑕疵，客服表示可以 申请15块补偿。(the user said that the product has some defects, and the customer service said that he could apply for a compensation of 15 yuan.) query 1: 亲能在上面热奶瓶吗？ (can it heat the bottle?) multimodal dialogue summarization task multimodal dialogue discourse parsing task 杯子底部需要平哦。(the bottom of bottle must be flat.) 底部不平就不可以呢。(it cannot be heated if bottom is uneven.) 亲就是这样的奶瓶。(that's the bottle.) response 1: query 3: response 2: query 2: answer : affirmative contingency: coordinate contingency: elaboration contingency: evaluation figure 1 : four segments of dialogue sampled from our jddc 2.1 corpus, corresponding to four tasks we define over the dataset. for each task, the output is marked in red. we show the second-level and third-level discourse relations for the dialogue discourse parsing task. note that these samples are truncated for demonstration purposes. with the widespread use of smartphones, taking screenshots and photos has been very convenient, and users often describe their issues by a combination of text and image when they communicate with customer service, which motivates us to explore multimodal dialogue tasks. before this work, we construct the jddc 2.0 corpus @xcite that is composed of multimodal dialogues where each dialogue session contains multiple pieces of text and at least one image, containing about 246 thousand dialogue sessions, 3 million utterances, and 507 thousand images, along with product knowledge bases and image category annotations. in this paper, on the foundation of jddc 2.0 @xcite , we build the jddc 2.1 corpus. over this dataset, we define four tasks, shown in figure 1 , including (1) a multimodal dialogue response generation task that predicts the current response given historical conversation records, (2) a multimodal query rewriting task that transforms multimodal queries into texts, aiming to reduce the difficulty of understanding multimodal utterances, (3) a multimodal dialogue discourse parsing task that converts a dialogue session into a discourse tree, aiming to analyze the discourse structure for dialogue sessions, and (4) a multimodal dialogue summarization task that generates a summary for a dialogue session, aiming to mine key informa-tion among utterances. note that the annotation of query rewriting, discourse parsing, and dialogue summarization is conducted with the same dialogue sessions, making it possible to explore the relationship between these tasks. so far as we know, jddc 2.1 is the first dataset with comprehensive annotations for all these above dialogue-related tasks. for all the tasks, we conduct experiments with text-only and multimodal baselines and verify the necessity of visual information for these tasks. ## related work visual dialogue dataset (visdial) @xcite first introduces visual contents into dialogues, in which the utterances are a group of questions and answers towards the corresponding image. this dataset focuses on understanding the given images. @xcite observes that, in social media, information for conversations around images is beyond what is visible in the image. they propose a new task called image-grounded conversation (igc) that aims to constitute conversations with the images as the grounding, where the objects in images may not be mentioned in the conversation. in other words, images in igc act as conversation topics. similar to igc, image-chat dataset @xcite is also an imagegrounded dialogue dataset, where the dialogue is performed based on a given emotional mood or style, which are key factors in engagingness @xcite . multimodal dialogue is the focus of this paper, which is different from the visual dialogue. we summarize the characteristics of multimodal dialogue as follows. (1) there may be more than one image for a multimodal dialogue session. (2) the images can be updated with the advance of dialogue. (3) the questions and answers can be either multimodal or monomodal. (4) knowledge bases may be useful for multimodal dialogue. (5) multimodal dialogue models sometimes need to clarify users' requirements with some dialogue strategies like rhetorical questions. as dialogue systems are widely deployed in the e-commerce domain, and nowadays, many online customer service robots have been put into use to provide 24-hour services to help customers solve various problems in the process of online shopping. there have been existing multimodal dialogue datasets in the e-commerce domain. @xcite build the multimodal dialogs dataset (mmd) that consists of over 150k dialogue sessions between shoppers and sales agents, with 84 dialog states and various conversation flows suggested by fashion retail experts. the dialogue scene in mmd is limited for the pre-sales guidance, while other scenes, such as payment, logistics, and aftersales maintenance, are not covered. the dataset of simmc 2.0 @xcite contains 11k dialogue sessions between customers and virtual assistants for situated and photo-realistic vr applications. similar to mmd, the target scene for simmc 2.0 is the pre-sales guidance. in fact, changes in scenes are quite frequent. for example, a dialogue system usually needs to solve customers' problems ranging from product selection, and payment, to logistics and distribution in a dialogue session. thus, in this paper, we collect the jddc 2.1 dataset that covers almost the complete process in e-commerce. in addition, to our knowledge, jddc 2.1 is the first dataset with joint annotations of multimodal dialogue response generation, multimodal query rewriting, multimodal dialogue discourse parsing, and multimodal dialogue summarization tasks. ## task definition we specify the definition of four tasks, namely the multimodal dialogue response generation task, the multimodal query rewriting task, the multimodal dialogue discourse parsing task, and the multimodal dialogue summarization task. the task of multimodal dialogue response generation is defined as: the multimodal query rewriting task aims to produce a text that covers the textual and visual information of the original multimodal query, which consists of two sub-tasks. the first one is to predict whether the multimodal query can be rewritten by a text without sacrificing critical information, and the latter one is rewritten query generation task that can be defined as: for multimodal dialogue summarization task aims at generating a condensed summary for a given dialogue session, which can be defined as: ## experiments in this section, we present experimental results for each task, human evaluations, error analyses, and case studies. ## conclusions and future work in this paper, we construct a real-scenario multimodal multi-turn chinese dialogue dataset named jddc 2.1 foot_2 . along with this dataset, we present four tasks, including the multimodal dialogue response generation task, the multimodal query rewriting task, the multimodal dialogue discourse parsing task, and the multimodal dialogue summarization task. for these tasks, we conduct experiments with text-only and multimodal baselines and show the necessity of visual information. however, there are still challenges in multimodal dialogue, such as understanding image details, effective modeling context information, long-tail questions, and refined dialogue strategy. in the future, we plan to further explore the multimodal joint modeling methods and approaches to using product knowledge bases and image category information, and we will provide more detailed annotation information on the dataset in terms of user emotions, user semantic analysis, and so on. furthermore, considering that all the annotations are shared with the same dialogue sessions, a promising research direction is to explore the interaction between these tasks, e.g., to investigate whether discourse structure across modality is helpful for understanding the multimodal dialogue, whether discourse information can indicate the important utterances, an so on.