R e S ee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue (2023)

Abstract: Incorporating visual knowledge into text-only dialogue systems has become a potential direction to imitate the way humans think, imagine, and communicate. However, existing multimodal dialogue systems are either confined by the scale and quality of available datasets or the coarse concept of visual knowledge. To address these issues, we provide a new paradigm of constructing multimodal dialogues as well as two datasets extended from text-only dialogues under such paradigm (ReSee- \texttt{WoW} , ReSee- \texttt{DD} ). We propose to explicitly split the visual knowledge into finer granularity (“turn-level” and “entity-level”). To further boost the accuracy and diversity of augmented visual information, we retrieve them from the Internet or a large image dataset. To demonstrate the superiority and universality of the provided visual knowledge, we propose a simple but effective framework ReSee to add visual representation into vanilla dialogue models by modality concatenations. We also conduct extensive experiments and ablations w.r.t. different model configurations and visual knowledge settings. Empirical, encouraging results not only demonstrate the effectiveness of introducing visual knowledge at both entity and turn level but also verify the proposed model ReSee outperforms several state-of-the-art methods on automatic and human evaluations. By leveraging text and vision knowledge, ReSee can produce informative responses with real-world visual concepts. Our code is available at https://github.com/ImKeTT/ReSee.

## introduction with the availability of large-scale datasets @xcite and pre-trained language models @xcite , dialogue generation develop rapidly in recent years. conducting effective linguistic ## multimodal dialogue datasets in this section, we introduce our framework for constructing multimodal dialogue datasets. the overall ## resee methodology we consider a simple approach to concatenate and to infuse multimodal information into plain dialogue models. as shown in figure 4 , we apply this approach to two transformer models with shared or separate encoder-decoder for dialogue responding. formally, we define our modeling task as: given the dialogue information ## conclusion in this paper, we present a paradigm for multimodal dialogue construction with two novel datasets and a multimodal dialogue responding framework re-see. we explicitly separate the visual knowledge into two aspects, using online searching or retrieving from large image corpora to construct accurate and diverse visual knowledge. transformer-based dialogue models with shared and separate encoderdecoder verify that provided visual knowledge promotes model capacity. further, we explore feeding multiple entity-level images and external document knowledge into models. by providing fine-grained visual knowledge on dialogues, we demonstrate dialogue models can substantially achieve better performance across different setups and domains.