A Framework for Vision-Language Warm-up Tasks in Multimodal Dialogue Models (2023)

Abstract: Most research on multimodal open-domain dialogue agents has focused on pretraining and multi-task learning using additional rich datasets beyond a given target dataset. However, methods for exploiting these additional datasets can be quite limited in real-world settings, creating a need for more efficient methods for constructing agents based solely on the target dataset. To address these issues, we present a new learning strategy called vision-language warm-up tasks for multimodal dialogue models (VLAW-MDM). This strategy does not require the use of large pretraining or multi-task datasets but rather relies solely on learning from target data. Moreover, our proposed approach automatically generate captions for images and incorporate them into the model’s input to improve the contextualization of visual information. Using this novel approach, we empirically demonstrate that our learning strategy is effective for limited data and relatively small models. The result show that our method achieved comparable and in some cases superior performance compared to existing state-of-the-art models on various evaluation metrics.

## introduction developing artificial intelligence (ai) that can converse naturally with humans is a primary goal of ai research. in particular, open-domain conversational agents that are not restricted to a specific domain have attracted considerable attention. many studies have adopted a pretraining approach using large datasets to improve the performance of these opendomain conversational agents @xcite . recent works have focused on multimodal opendomain conversational agents that consider visual information in addition to textual information for dialog generation. this approach utilizes visual information to help understand the context of a image turn 1 i'm worried about this race (anxious) turn 2 all these bike races are happening near my street. (dull) turn 3 i'd be worried about accidentally hitting one of them (anxious) to build such multimodal open-domain conversational agents, most previous studies have considered multi-task learning or have utilized pretrained models using large-scale data beyond the target data @xcite . this allows models trained with only text information to accept visual information. however, the requirement of collecting additional datasets is restrictive, and pretraining with additional datasets is inefficient in terms of time and resources. in this study, we propose a method to align text and images using only the target data to address these issues. we experimentally evaluated the effectiveness of the proposed method with limited data or smaller models. the proposed method utilizes only target data to align images and texts. by automatically generating captions for the images and adding them as input to the model, our approach enables a text-based pre-trained model to process image information more effectively. this differs notably from existing multimodal open-domain models that only receive image and utterance information. we propose visionlanguage warm-up tasks for multimodal dialogue models (vlaw-mdm) to effectively integrate information from images, captions, and context data. to construct the framework, we incorporated four warm-up tasks based on existing multimodal pretraining models @xcite @xcite . these tasks include generation captioning (gcp), image swapping (isp), masked region modeling (mrm), and masked language modeling (mlm). they can be applied using only target data without any additional data required. these warm-up tasks enable the model to learn the associations between images and utterances. the construction of this framework was inspired by the generative framework used by @xcite , which we reformulate into a multimodal architecture suitable for the purposes of this research. we apply our warm-up framework to the popular sequence-to-sequence models bart and blender-bot. for blenderbot, we used a smaller model (400m) than that (2.7b) adopted in the multi-modal blenderbot (mmb) @xcite . this allowed us to explore the performance of our proposed framework on smaller versions of the model and to validate the framework against a relatively large model with additional training data. we used the image-chat dataset @xcite to evaluate the effectiveness of our proposed framework. the data were structured as shown in the main contributions of this study are summarized as follows: ## related work vision and language tasks. the integration of language and vision is a topic of active research that traditionally includes tasks such as image captioning and visual question answering (vqa) @xcite @xcite @xcite @xcite . image captioning tasks focus on generating appropriate descriptions of a given image. major datasets include coco captions @xcite and flickr30k @xcite . these datasets of images covering various topics provide an ideal benchmark for assessing model's ability to understand complex content in an image and express it in natural language. sequence-to-sequence structures are the most common method to process these datasets @xcite @xcite . the vqa task @xcite requires image recognition and factual verification of text content. it evaluates the ability of a model to generate accurate answers to questions related to a given image. as a natural extension of this work, a method has been proposed to generate questions that can capture a user's attention based on a given image. however, these methods involve the limitation that the conversation usually ends in a single turn. to address this, visual dialog @xcite extends this with a continuous question-answer scenario. however, this approach does not provide a direct way to evaluate whether users experience conversations as interesting and engaging. to address this, multimodal multi-turn dialog datasets (mostafazadeh et al., 2017; shuster et al., 2020a) have been proposed. in particular, image-chat (shuster et al., 2020a) supports multi-turn dialog based on a single image. each utterance contains style information to allow a model to learn different styles of conversation. hence, image-chat is an ideal dataset for training multimodal open-domain agents and enables more engaging dialog generation. multimodal representation learning. utilizing the weights of existing models trained on a single modality and fusing them together is a common strategy for multimodal learning @xcite @xcite @xcite . many studies have adopted this approach to reuse models pretrained on a single modality for multimodal representation learning. to process images and text together, @xcite uses multi-task training, in which a model learns by bundling multiple tasks that are related to a given target task. for this purpose, @xcite includes 12 subtask sets, which allows it to perform multiple tasks on large datasets. in some cases, multimodal representations have been learned from multimodal datasets @xcite . however, multimodal pretraining approach has generally not been performed with data in the target domain. one potential solution is to employ domain-adaptive pretraining by using data related to the domain of the target data @xcite . this method enables a model to adapt more effectively to a specific domain. however, domain-adaptive pretraining also utilizes data related to the target domain and does not provide a pretraining methodology for specific target data. as a solution, @xcite proposed a task-specific vision-language pretraining framework for multimodal aspect-based sentiment analysis, which realizes target data-specific pretraining on multiple tasks instead of a single task. ## methodology in the present work, we adopt blenderbot as a backbone model. the overall architecture is illustrated in figure 1 . in this section, we describe the operation of the entire framework. first, we describe how we process the images used as input to the model. in particular, we discuss our approach to extract features from an image and generate captions. then, we describe how the encoder is extended to handle multimodal inputs and how the decoder generates utterances from the information received from the encoder. finally, we describe the warm-up tasks that comprise the framework. ## conclusion we have proposed vlaw-mdm as a methodology for training multimodal open-domain agents using only target data to obviate the need for large amounts of data for pretraining or multiple tasks. the experimental results have shown that even with limited data, a model pretrained from a single modality can effectively process multimodal information. furthermore, our proposed approach outperformed existing models in terms of f1 score on the image-chat dataset and outperformed mmb in terms of f1 and bleu-4 scores despite its smaller size. in future work, we plan to explore extensions to this framework.