Exploring Prompt-based Multi-task Learning for Multimodal Dialog State Tracking and Immersive Multimodal Conversation (2023)

Abstract: With the rise of the metaverse, immersive multimodal conversation has attracted more and more researchers’ attention. Multimodal contexts will become more important for human-computer interaction in the metaverse, especially in shopping domain. Unlike traditional conversation tasks, immersive multimodal conversation has challenges such as multimodal ambiguous candidate identification and multimodal coreference resolution, which makes it more difficult to dialog state tracking and response generation, as described in SIMMC 2.1 challenge, a part of DSTC11. In particular, as the number of objects in the scene increases, the difficulty will increase dramatically. We proposed a prompt-based multi-task learning Encoder-Decoder, in which different subtasks use different prompts to make the model tend to focus on the current subtask. We achieve the winner in ambiguous candidates indentification and runner-up in multimodal coreference resolution (MM-Coref), multimodal dialog state tracking (MM-DST) and assistant response generation. Our code and model are made publicly available at https://github.com/scutcyr/dstc11-simmc2.1-scut-bds-lab.

## introduction with the rise of the metaverse @xcite and virtual reality (vr), immersive multimodal conversation has attracted more and more researchers' attention. unlike traditional conversation tasks, immersive multimodal conversation has challenges such as multimodal ambiguous candidate identification and multimodal coreference resolution, which makes it difficult to dialog state tracking and response generation. to this end, facebook released multimodal conversational dataset (simmc 2.1) and proposed the simmc 2.1 challenge @xcite , in which the virtual assistant shares the same scene with the user. in the past, the simmc 1.0 @xcite ) challenge provided the controllable and sanitized multimodal contexts, while the simmc2.0 @xcite challenge provied the cluttered and closer-to-real-world multimodal contexts, in which there is no clear correspondence between textual context and objects in the scene. on the basis of the simmc 1.0 and simmc 2.0, simmc 2.1 proposed a new subtask (ambiguous candidate identification) in order to attract more attention on the key challenge of fine-grained visual disambiguation, as shown in figure 1 . different from the simmc 2.0, simmc 2.1 has been annotated with additional labels (i.e. identification of all possible 1 referent candidates given ambiguous mentions). in particular, the number of objects in a scene ranges from 6 to 141. each object is different due to its visual or non-visual attributes. there are multiple objects with one same attribute in the scene, but the number of user-mentioned objects is less than the count in the scene, which makes the virtual assistant unable to identify the specific object that the user refers to, thus causing ambiguity. the simmc2.1 challenge has four different subtasks, which are ambiguous candidates indentification (aci), multimodal coreference resolution (mm-coref), multimodal dialog state tracking (mm-dst) and assistant response generation. due to the introduction of aci, the other three subtasks are affected accordingly. in addition, ambiguous candidates and multimodal coreferential objects do not appear at the same time based on statistical analysis of the dataset. the state-of-the-art method in the previous simmc challenge @xcite adopt bart and joint-learning approach on all subtasks. however, the performance of the original framework will be limited after adding the ambiguous candidates indentification task. in addition, the model also ignores the ground truth apis in terms of response generation performance. modeling mm-dst tasks as sequential prediction tasks is also easy to cause instability in prediction. for example, the model may forget to generate "<@xmath0. in order to consider both the performance and robustness of the model, we propose a prompt-based multi-task learning transformer framework to address the above problems. in particular, aci, mm-coref and mm-dst share the same prompt, while the assistant response generation task uses another set of prompts alone, which will make full use of groundtruth apis when generating responses. our model was declared as the winner of the subtask 1 (ambiguous candidates indentification) with 70.50% ambiguous object identification f1. moreover, our model was declared as the runnerup in the official evaluation on all other subtasks, in which we achieved 80.28% coreference f1, 92.66% slot f1, 97.75% intent f1 (rank #1) and 0.3650 bleu-4. ## method in this section, we demonstrate the implementation of the proposed pmtled for solving each subtask. the architecture for subtasks 1, 2 and 3 is shown in figure 3 (a), and the architecture for subtask 4 is shown in figure 3 (b) . we first introduce the input of the model and the model architecture. then we elaborate on objectives of each subtask and auxiliary tasks. ## conclusion and future work in this paper, we proposed a prompt-based multitask learning method for multimodal dialog state tracking and immersive multimodal conversation. the proposed model used fixed prompt for multimodal dialog state tracking and dynamic prompt for immersive multimodal conversation based on transformer framework. experiments demonstrated that the proposed fixed prompt and dynamic prompt are effective on jointly fine-tuning the model for the simmc 2.1 challenge. in the future, we will continue exploring the design of the prompt, promoting the model to apply more knowledge as much as possible, e.g. the mentioned objects of current turn of the assistant, the scene image and etc. in addition, the current version of the model depends on domain information, which makes it difficult to apply to other scenarios. further research needs to solve the domain dependency.