Contrastively Pretrained Vision-Language Transformers and Domain Adaptation Methods for Multimodal TOD Systems (2023)

Abstract: The Situated Interactive MultiModal Conversations (SIMMC2.1) Challenge 2022 is hosted by the Eleventh Dialog System Technology Challenge (DSTC11). This is the third consecutive year multimodal dialog systems have been selected as an official track of the competition, promoted by the continued interest in the research community. The task of SIMMC is to create a shopping assistant agent that can communicate with customers in a virtual store. It requires processing store scenes and product catalogs along with the customer’s request. The task is decomposed into four steps and each becomes a subtask. In this work, we explore the common approaches to modeling multimodality and find the method with the most potential. We also identify a discrepancy in using pretrained language models for dialog tasks and devise a simple domain-adaptation method. Our model came in third place for object coreferencing, dialog state tracking, and response generation tasks.

## introduction task-oriented dialog system refers to a system that communicates with users with a specific purpose, such as restaurant recommendation or accommodation reservation. the system should be able to analyze the user's utterance to track the dialog state and generate a response accordingly to fulfill the user's purpose. recent advances in pretrained language models and new dialog datasets such as multiwoz have accelerated the research on task-oriented dialogs @xcite . to this end, situated interactive multimodal conversations (simmc) challenge was proposed to introduce multimodality to the task-oriented dialog tasks @xcite . the challenge consists of a virtual shopping scenario in the fashion and furniture domain. each dialog is based on up figure 1 : the simmc2.1 dataset consists of the dialogues between a user and a clerk, with an image of the virtual store's scene and the metadata for every object that appears in the scene. to achieve high performance on the given four subtasks, it is important to utilize all possible modalities with the most adequate method. to two virtual scenes, and the user interacts with the system to purchase items in the corresponding domain. the system is required to process user utterances, previous dialogues, images of the scenes, and metadata files for the objects appearing in the scenes to find the adequate belief state and generate a response. an example of dialog is given in figure 1 , where the user seeks two pairs of trousers to purchase. as the dialogues are grounded on virtual scenes, the system should not only have a powerful language understanding but also should be able to process images and fuse the two modalities. in the eleventh dialog system technology challenge (dstc11), simmc2.1 was held as the first track for competition. the simmc2.1 challenge decomposes the requirements of multimodal taskoriented systems into the following four subtasks: 1) ambiguous candidate identification, 2) multimodal coreference resolution, 3) multimodal dialog state tracking, 4) multimodal dialog response generation. the only change from simmc2.0 @xcite is the first subtask, where the binary classification task developed to be a more challenging candidate identification task. past studies have proposed various designs for solving the simmc task. for instance, huang @xcite proposed a model based on a vision-language transformer model and condensed various metadata and visual data of objects into a single token. lee @xcite utilized contrastive learning to pretrain a dual encoder model to project image data and language data into the same semantic space. son @xcite converted image data to a natural language form with multiple attribute classifiers and leveraged the power of a large pretrained language model for better language generation results. likewise, approaches vary widely on how to process data from different modalities. in this study, we identify three methods that are commonly used to model multimodality which is; monolithic vision language transformers, dual encoding vision language transformers, and languagecentric multimodal transformers. we select a representative model from each category and conduct experiments on simmc2.1 to compare the performance of each method. we also develop a retrievaloriented approach for obtaining the meta-data of individual objects, which substantially improves the precision of the predicted attributes for each object. we find it unsuitable to fine-tune pretrained language models for the simmc task. most pretrained language models pretrain on large text datasets that are crawled from the internet, which are predominantly web pages and documents. however, dialogues have different characteristics compared to typical text data because the vocabulary for colloquial style differs from written text @xcite . to be specific, spoken language tends to have frequent coreferencing and ellipsis, which require earlier contexts to understand. to this end, we experiment with several domain-adaptation methods based on pretraining to alleviate the problem. ## methods we present three representative methods for solving multimodal tasks and a pretraining method to enhance performance by adapting large pretrained language models to the dialog domain. ## conclusion in this work, we have identified 3 common ways to model multimodality and conducted experiments to find the best fit for multimodal task-oriented dialog systems. despite that we found the dual vision-language transformer to have higher performance over other methods we did not set specific criteria for each method to be fairly evaluated such as the size of parameters and number of training steps. we also investigated domain-adaptation methods to enhance language understanding and generation in the dialog domain, where we found the noun-prioritized masking effective. it should be interesting to leverage part-of-speech tags for domain-adaptation to other domains, which could be material for future work.