P a CE : Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts (2023)

Abstract: Perceiving multi-modal information and fulfilling dialogues with humans is a long-term goal of artificial intelligence. Pre-training is commonly regarded as an effective approach for multi-modal dialogue. However, due to the limited availability of multi-modal dialogue data, there is still scarce research on multi-modal dialogue pre-training. Yet another intriguing challenge emerges from the encompassing nature of multi-modal dialogue, which involves various modalities and tasks. Moreover, new forms of tasks may arise at unpredictable points in the future. Hence, it is essential for designed multi-modal dialogue models to possess sufficient flexibility to adapt to such scenarios. This paper proposes PaCE, a unified, structured, compositional multi-modal dialogue pre-training framework. It utilizes a combination of several fundamental experts to accommodate multiple dialogue-related tasks and can be pre-trained using limited dialogue and extensive non-dialogue multi-modal data. Furthermore, we propose a progressive training method where old experts from the past can assist new experts, facilitating the expansion of their capabilities. Experimental results demonstrate that PaCE achieves state-of-the-art results on eight multi-modal dialog benchmarks.

## introduction enabling seamless communication between humans and machines is a long-standing goal of artificial intelligence research. the recent emergence of chatgpt 1 has increased confidence in the potential for achieving this goal. beyond the use of textual language as a unique interface between humans and machines, perceiving and utilizing multi-modal information, especially visual information, has be-* equal contribution. hi, if you're traveling to asia, the palace museum is worth seeing, you can get to see many ancient artifacts there. wow, that sounds interesting to me. could you tell me more about some of the stories related to the palace museum? the palace museum is located in the forbidden city, which was the imperial palace of the ming and qing dynasties in china. it is called the "forbidden city" because ordinary people were not allowed to enter without permission. is the place in this picture worth going to? it looks mysterious, could u give me some introduction about it ? sure ! the temple of heaven is also a culturally significant site, known for its historical use as a location for imperial sacrifices in ancient china. come a crucial capability known as multi-modal dialogue @xcite . to facilitate the research on multi-modal dialogue, plenty of specific tasks and datasets have emerged in the community @xcite @xcite . however, the overall quantity of data is still limited. furthermore, multi-modal dialogue presents a greater challenge compared to traditional text-only dialogue track @xcite @xcite , as it involves the integration of various modalities and more intricate task scenarios. as shown in figure 1 , the central tasks of multi-modal dialogue include multi-modal intent classification @xcite , multi-modal dialogue retrieval @xcite , multi-modal dialogue state tracking @xcite , and multi-modal response generation @xcite . despite pre-training having become the consensus for multi-task learning in machine learning @xcite @xcite , the research on pre-training models for multi-modal dialogue is an area that is yet to be fully explored. in this paper, we focus on building pre-trained models of multi-modal dialogue. a key challenge is to unify different modalities and task forms, and make the best use of existing multi-modal dialog and non-dialog data. a recent popular trend on textual tasks is to build unified pre-trained foundation models by multi-task learning, e.g., t5 @xcite . however, it attempts to mix all tasks learned from scratch thus is difficult to control the learning process, which is a completely black box. although the mixture-of-experts (moe) @xcite architecture attempts to select independent experts for each input sample through token-level routing, it lacks specific semantics, i.e., it is entirely unknown what the experts are responsible for. we hope to find a new way to handle many multi-modal dialog tasks simultaneously and combine existing concrete skills to learn new tasks more efficiently. to this end, we propose pace, a unified multi-modal dialogue pre-training framework with progressive and compositional experts. first, we decompose complicated multi-modal dialogue into fundamental sub-capabilities that could be learned with specific data. different from traditional moe, each expert in pace is tailored to one specific fundamental sub-capability of multi-modal dialogue, including caption, context, image, grounding and generation. second, we propose a progressive pre-training strategy to evolve the model by controlling the combination of experts in different pre-training phases. specifically, in stage i, we first train on multi-modal non-dialogue data to obtain caption, image, and grounding experts. in stage ii, we train the context expert, which is guided by the caption expert on multimodal dialog data to learn the dependencies in context. furthermore, a dialogue generation expert is derived by adding a response generation task based on the previously learned experts. third, for pre-training pace, we collect a multi-modal dialog corpus with 1.4 million dialogs and a multi-modal non-dialog corpus with 4 million samples. once mmconv mmconv photochat image-chat simmc mmdialog photochat simmc previous sota pace (response genera,on) (dialog state tracking) (intent classifica,on) (intent classifica,on) (response genera,on) (text retrieval) (image retrieval) 10.4 15.2 18.0 39.2 33.1 34.1 32.2 44.7 75.5 77.6 50.3 51.9 58.9 63.8 (dialog state tracking) 96.3 97.1 the pre-training of pace is finished, we can flexibly select different capability experts to solve a specific downstream task. as illustrated in figure 2 , pace achieves stateof-the-art performance across a broad range of multi-modal dialogue benchmarks spanning four diverse downstream tasks, i.e., multi-modal intent classification, multi-modal dialogue retrieval, multi-modal state tracking, and multi-modal response generation this demonstrates that pace not only possesses a flexible model architecture but also exhibits adaptable training methodologies, resulting in remarkable performance. ## related work pre-trained vision-language models the pretraining paradigm, with its successes in natural language processing @xcite , has sparked a revolution in multimodal learning. vilbert @xcite was the first work to adapt the bert-like architecture for visual-language modeling, allowing for learning joint representation of images and texts. vilt @xcite constructed the vision module in the same way as the text module with a unified transformer @xcite , eliminating the need for resource-intensive image feature extraction and significantly accelerating the model. clip @xcite employed contrast learning to directly align images with natural language texts, eliminating the constraints of predefined image categories. align @xcite and florence @xcite further generalized this idea on noisier but larger image-text pairs. these models have demonstrated the ability to learn strong image and text representations for cross-modal alignment tasks. in addition, a number of models @xcite @xcite @xcite employed auto-regressive models to model the association between images and texts, using a unified generation approach to construct the task in an end-toend manner. although pre-trained vision-language models have shown promising results, they mainly focus on caption texts which are intrinsically different from human conversations @xcite . to our best knowledge, the proposed pace model is the first multi-modal dialogue pre-training model. multi-modal dialogue modeling numerous advanced works have been proposed along with the development of multi-modal dialogue datasets @xcite @xcite @xcite @xcite . several dialogue modeling works @xcite have been conducted to improve the performance of conversational agents in image-grounded dialogue. @xcite proposed a dual-encoder model that utilized object labels to encode image features so as to perform a dialogue-based image retrieval task. afterward, researchers @xcite explored enriching textual expressions of generated dialogue responses through associative vision scenes. for textual response tasks, @xcite proposed a multi-modal dialogue generation model based on seq2seq architecture, which was proved to be superior to the textual seq2seq model. @xcite proposed a joint multimodal encoder-decoder model to incorporate visual inputs. however, the above models have demonstrated success in specific sub-tasks with a particular dataset, which cannot meet the requirements of a wide range of multi-modal dialogue tasks. to address this challenge, we propose a unified multi-modal dialogue pre-training model based on a divide-and-conquer strategy, which can combine different experts to complete a series of tasks. ## pre-training data construction in this paper, we collect both multi-modal nondialogue and multi-modal dialogue data for pace pre-training. the total statistics of our collected pre-training corpora are shown in ## pre-training method multi-head self-attention (a) pre-training stage i (l-f) x image text matching multi-head self-attention image context matching ffn ffn ffn ffn ffn context caption image grounding generation ffn ffn ffn ffn ffn context caption image grounding generation multi-head self-attention response generation modeling ffn ffn ffn ffn ffn context caption image grounding generation (b) pre-training stage ii (c) pre-training stage iii (l-f) x (l-f) x multi-head self-attention ffn ffn ffn ffn ffn context caption image grounding generation f x text context alignment patch embeddings patch embeddings word embeddings a dog [mask] and a cat are playing on [mask] snow . word embeddings patch embeddings word embeddings hi, i am traveling [mask] norway now . … … cool ! did you take your puppies [mask] you . two dogs [mask] running on [mask] snow . hi, i am traveling on norway now . … … cool ! did you take your puppies with you . sure, here they are. ## experiments 5.1 downstream datasets to comprehensively evaluate our pace, we conduct extensive experiments on seven datasets belonging to four downstream tasks. ## conclusion in this paper, we proposed pace, a unified, structured, compositional multi-modal dialogue pretraining framework, which adopted a divide-andconquer strategy. we first break down the complicated multi-modal dialogue generation task into several sub-capabilities, which could be learned in an easier way. then, the solutions to the subcapabilities were combined to obtain an effective and efficient solution to each downstream multimodal dialogue task. experimental results on eight benchmark datasets demonstrated that pace achieved new state-of-the-art performances.