M ulti DM - GCN : Aspect-guided Response Generation in Multi-domain Multi-modal Dialogue System using Graph Convolutional Network (2020)

Abstract: In the recent past, dialogue systems have gained immense popularity and have become ubiquitous. During conversations, humans not only rely on languages but seek contextual information through visual contents as well. In every task-oriented dialogue system, the user is guided by the different aspects of a product or service that regulates the conversation towards selecting the product or service. In this work, we present a multi-modal conversational framework for a task-oriented dialogue setup that generates the responses following the different aspects of a product or service to cater to the user’s needs. We show that the responses guided by the aspect information provide more interactive and informative responses for better communication between the agent and the user. We first create a Multi-domain Multi-modal Dialogue (MDMMD) dataset having conversations involving both text and images belonging to the three different domains, such as restaurants, electronics, and furniture. We implement a Graph Convolutional Network (GCN) based framework that generates appropriate textual responses from the multi-modal inputs. The multi-modal information having both textual and image representation is fed to the decoder and the aspect information for generating aspect guided responses. Quantitative and qualitative analyses show that the proposed methodology outperforms several baselines for the proposed task of aspect-guided response generation.

## introduction conversational systems have become ubiquitous in our everyday lives. previous research suggests that the conversational agents need to be more interactive and informative for building engaging systems @xcite . these research indicates that engaging conversations include visual cues (e.g., a video or images) or audio cues (e.g., tone, the pitch of the speaker). information contained in these cues is often integral for the conversation. in figure 1 , we show an example of a conversation where the visual cues in the form of images are crucial for better understanding and interactive dialogue between the agent and the user. the appropriate responses to the user queries are highly dependent on the visual information pertaining to the different aspects of the various images in the conversation. thus, it is natural to conclude that a conversational agent would be more effective if the visual information were part of its underlying conversational model. multi-modality in goal-oriented dialogue systems figure 1 : examples from the multi-domain multimodal dialogue(mdmmd) dataset @xcite for the fashion domain has established the significance of visual information for effective communication between the user and agent. inspired by their works, we take a step forward by creating a multi-modal aspect guided response framework for a multi-domain goal-oriented dialogue system. from figure 1 , it can be observed that visual information of the aspects encourages improved communication and informative response generation by the agent with regards to the user queries. in this paper, we propose the task of generating informative and interactive responses guided by the aspect information in a multimodal dialogue system. firstly, we create a high quality multi-modal conversational dataset. thereafter, we present a multi-modal graph convolutional network (gcn) that incorporates information from both textual and visual modalities to generate the aspect-guided responses. we aim to create a generalized response generation framework for a multi-domain multimodal dialogue system that is informative, interesting, aspect-guided, and logical. hence, the main contributions of this work are: (i) we propose the task of aspect-guided response generation for the interactive and informative responses in a multimodal dialogue system. this is the first attempt to incorporate aspect information in the multi-modal dialogue systems to the best of our knowledge. ## related work uni-modal dialogue systems the effectiveness of deep learning has shown significant progress in dialog generation. deep neural frameworks, as shown in the @xcite , are very effective in modeling conversations. the hierarchical encoder-decoder system was studied in @xcite @xcite to preserve the dependencies among the utterances in dialogue. recently, memory networks @xcite @xcite @xcite @xcite have been investigated to capture the contextual information in dialogues for generating responses. in taskoriented dialogues, hierarchical pointer networks (raghu and gupta) have been used to generate the responses. with the release of the task-oriented dialog dataset, such as multiwoz , a few works @xcite have emerged that operate in a multi-domain dialogue setting. the meta-learning approach @xcite has been implemented on the various datasets to improve the domain adaptability for generating responses. ## dataset in this section, we describe the procedure of creating multi-modal dialogue data. ## methodology for the proposed task, we assume that the aspect term information will be provided for the response to be generated. as different aspects are extremely subjective in a goal-oriented system, hence the responses are majorly dependent upon the respondent. therefore, there can be several potential responses possible for a given input. because of this subjectivity in goal-oriented systems, we like to focus on solving the task of generating responses with the desired aspect information. problem definition: our current work addresses the task of aspect guided response generation in a multi-modal goal-oriented dialog system conditioned on the conversational history having both textual and visual information. to be more specific, given an utterance u @xmath0,1 , w k,2 , ..., w k,n ), a set of images i @xmath1,1 , i k,2 , ..., i k,j ), and a conversational history h @xmath2, i 1 ), (u 2 , i 2 ), ..., (u k-1 , i k-1 )) and the aspect term v a the task is to generate the next textual response @xmath3, y 2 , ....., y n ), where n and n are the given input utterance and response length, respectively. ## experimental details in this section we present the details of the experimental setup and evaluation metrics. implementation details all the implementations were done using the pytorch foot_1 framework. for all the models including baselines, the batch size is set to 32. the utterance encoder is a bidirectional gru with 600 hidden units in each direction. we use the dropout @xcite with probability 0.45. during decoding, we use a beam search with beam size 10. the model is initialized with the parameters chosen randomly using a gaussian distribution with the xavier scheme @xcite . the hidden size for all the layers is 512. amsgrad @xcite is used as the optimizer for model training to mitigate the slow convergence issues. we use uniform label smoothing @xmath4.1 and perform gradient clipping when the gradient norm is above 5. we use 300-dimensional word-embedding initialized with glove @xcite embedding pre-trained on twitter. we consider the previous 2 turns for the dialogue history, and the maximum utterance length is set to 50. for image representation, fc6(4096 dimension) layer representation of the vgg-19 @xcite , pre-trained on imagenet is used. ## results and discussion in this section we report the evaluation results along with the necessary analysis and discussions on these. automatic evaluation results: evaluation results using automatic evaluation metrics are provided in ## conclusion and future work our current work emphasizes on the task of generating aspect-guided responses in a multi-modal dialogue system. we create a large scale task-oriented mdmmd dataset comprising of dyadic dialogues. the dataset comprises of three different domains, such as restaurant, electronics, and furniture. we develop a gcn based method to capture the textual representation, while we use vgg-19 for image representation. the context encoder captures the multi-modal information from the utterances. the representation from the context encoder along with the aspect vector is fed to the decoder for generating the aspect-guided responses. experimental results show that our proposed methodology outperforms the baseline models in the case of both automatic and human evaluation metrics. in future along with enhancing the architectural design of our proposed methodology, we would also like to investigate methods for image retrieval for complete multi-modal response generation. furthermore, we would extend our method to deal with multiple aspects present in an utterance and generate the responses accordingly. chatbot, sponsored by serb, govt. of india (imp/2018/002072). asif ekbal acknowledges the young faculty research fellowship (yfrf), supported by visvesvaraya ph.d. scheme for electronics and it, ministry of electronics and information technology (meity), government of india, being implemented by digital india corporation (formerly media lab asia).