Going beyond Imagination! Enhancing Multi-modal Dialogue Agents with Synthetic Visual Descriptions (2024)

Abstract: Building a dialogue agent that can seamlessly interact with humans in multi-modal regimes, requires two fundamental abilities: (1) understanding emotion and dialogue acts within situated user scenarios, and (2) grounding perceived visual cues to dialogue contexts. However, recent works have uncovered shortcomings of existing dialogue agents in understanding emotions and dialogue acts, and in ground- ing visual cues effectively. In this work, we investigate whether additional dialogue data with only visual descriptions can help dialogue agents effectively align visual and textual features, and enhance the ability of dialogue agents to ground perceived visual cues to dialogue contexts. To this end, in the absence of a suitable dataset, we propose a synthetic visual description generation pipeline, and con- tribute a large-scale synthetic visual description dataset. In addition, we propose a general training procedure for effectively leveraging these synthetic data. We conduct comprehensive analyses to evaluate the impact of synthetic data on two benchmarks: MELD and IEMOCAP. Our findings suggest that synthetic visual descriptions can serve as an effective way to enhance a dialogue agents’ grounding ability, and that the training scheme affects the extent to which these descriptions improve the agent’s performance.

## introduction there have been impressive advances in large-scale vision and language models (vlms) in performing multi-modal tasks, such as visual question answering (vqa) and image captioning @xcite @xcite . while vlms are powerful general-purpose models for a wide range of tasks, most state-of-the-art vlms still struggle with providing real-world, situated multi-modal assistance @xcite . frame 1 frame 2 ross: or, we could go to the bank, close our accounts and cut them off at the source. ross raises his eyebrows, leans a little bit forward, and points to a direction. chandler looks towards ross, speaking in an excited, approved tone. ## synthetic data generation to appropriately understand an interlocutor's emotions and dialogue acts, vlm-based dialogue agents must ground perceived visual cues within dialogue contexts. we hypothesize that current vlms are limited by a large gap between different modalities, which affects their ability to ground visual cues to dialogue contexts. however, the annotation that links visual cues and dialogue contexts is missing from existing widely-used datasets (e.g., meld @xcite ). in this section, we investigate whether additional visual descriptions can help dialogue agents, and to what extent synthetic data can be used to bridge the gap between visual cues and dialogue contexts. we can easily have access to large amounts of dialogue contexts, but it is hard to obtain the corresponding images or videos. in the absence of suitable multimodal datasets with the grounding annotations, we propose a visual description generation approach in the rest of this section. we then propose a training procedure ( § 3) for leveraging this synthetic data to improve the performance of multi-modal dialogue agents on the tasks of dialogue act and emotion prediction. problem formulation and notation. given a multi-turn dialogue @xmath0, . . . , u m } consisting of a sequence of utterances u i in plain text, our goal is to prompt chatgpt to generate synthetic visual description v ′ i for each utterance. we will get a synthetic dataset in which each of the utterances is paired with a synthetic visual description instead of a image. this synthetic dataset, augmented with the visual descriptions, will be used in training for reducing the modality gap, as explained below; see figure 2 for an example. ## fine-tuning a vlm-based agent using synthetic data in this section, we propose a methodology for leveraging the synthetic data produced as explained in section 2. one intuitive way is to combine synthetic data with real data for training. however, as there is a large gap in both modalities and patterns between real images and synthetic visual descriptions, a straightforward concatenation of real and synthetic dataset would not be the best choice. we therefore propose a multi-stage training framework, which trains the dialogue agent with synthetic and real data separately, followed by a knowledge distillation training stage @xcite . specifically, we choose the state-of-the-art vlm model, llava-v1.5 @xcite as the backbone of our system, which integrates the visual encoder of clip @xcite with the language decoder vicuna @xcite . fine-tuning with synthetic data. suppose we have a synthetic training dataset of dialogues d s , where each dialogue fine-tuning with real data. the goal of finetuning with real data is to adapt the dialogue model to the real multi-modal situation. we have a real dataset d r containing a set of multi-modal dialogues, where each dialogue @xmath1, v 1 , y 1 ), ..., (u m , v m , y m )} has m utterances (u), corresponding images (v) and output labels (y). unlike synthetic data fine-tuning, the clip and vicuna module within the llava-v1.5 will be used to process visual images and dialogue contexts collaboratively. this process will yield a fine-tuned adapter θ r for the real data. knowledge distillation. the distillation training procedure aims to transfer the "imagination" ability learnt from the synthetic data to enhance dialogue agents in grounding visual cues to dialogue contexts in multi-modal settings. we conduct the knowledge distillation procedure on the fine-tuned adapters θ s and θ r by applying the kldivergence @xcite ) regularization in three different settings, as follows. ## experiments in our experiments, we aim to investigate the following two research questions: (1) how effectively do existing vlm-based agents comprehend emotions and dialogue acts from humans?, and ( 2 ) to what extent can synthetic visual descriptions enhance agents' multi-modal capabilities in understanding emotions and dialogue acts. ## related work visual dialogue. the visual dialogue task was proposed by @xcite . it requires an agent to answer multi-round questions about a given image, similarly to visual question answering (vqa) @xcite @xcite . previous work @xcite @xcite @xcite @xcite focused on developing different attention mechanisms to model the interactions among image, question and dialogue history @xcite . with the rapid development of large-scale vision-language models (vlms) @xcite @xcite @xcite , recent work focuses on building unified models that can handle multiple tasks. however, most models are still unable to support situated interaction with humans in real scenarios, especially capturing human emotions and dialogue acts, and grounding to their dialogue contexts. learning from synthetic data. there has been some work on learning from synthetic data for dialogue systems @xcite @xcite @xcite . synthetic data are easy to generate, and are particularly useful for providing detailed labelling to reduce human labor, such as dialogue acts @xcite , knowledge injection @xcite or simulating dialogues in new scenarios, such as the rapid generation of a sequence of qa from documents @xcite . however, these works mainly focus on plain text dialogues, rather than multi-modal dialogues. we propose a novel framework to utilize synthetic data to address this gap, and thereby enhance the abilities of multi-modal dialogue agents on the task of emotion and dialogue act classification. ## conclusion our work demonstrates the potential of synthetic visual descriptions to improve the performance of dialogue agents, particularly in understanding emotions, dialogue acts and grounding visual cues to dialogue contexts. by introducing a novel synthetic visual description generation pipeline and a largescale dataset, along with an effective training procedure, we have taken a crucial step towards overcoming the limitations of multi-modal dialogue agents. the positive outcomes observed in our experiments highlight the importance of appropriate training schemes to fully leverage synthetic data, pointing towards a promising direction for future research.