Multi-Task Learning for Ambiguous Candidate Identification with Pre-trained Model (2023)

Abstract: Recently, research using multimodal datasets containing image and text information has been conducted actively. One of them is the SIMMC2.1 dataset. It is a more complicated dataset than answering a conversation using only text because it should predict an answer after understanding the relationship between images and text. Therefore, there are limitations to answering a conversation only using text-based models such as BERT or GPT-2, so models with both image and language understanding abilities should be considered. We propose a new model that is effective for the ambiguous candidate identification task in DSTC11 SIMMC2.1 Tark. It consists of a simple pipeline model structure, which has two steps. The first step is to check whether there is ambiguity in the current user utterance, and the second step is to extract objects mentioned in the ambiguous utterance of the user. We suggest a new learning framework with a pre-trained image model and text model that is effective for the ambiguous candidate identification task. Experiments show that the proposed method can improve the model performance, and our model achieved 3rd place in sub-task 1 of the SIMMC2.1 track.

## introduction conversations with multiple contexts, and multimodal components, have been actively explored @xcite @xcite . when developing the conversational agent in the real world, visual information is essential for effective conversation agents so that they can easily adapt to user situations. in this paper, we present our ambiguous candidate identification model, which was submitted to the simmc2.1 challenge @xcite . we focus on the ambiguous candidate identification task of simmc2.1, * these authors contributed equally to this work among other tasks. it is the task of detecting ambiguity in conversations and extracting objects mentioned in the user utterance predicted as ambiguous. for example, ambiguities may occur for several reasons, such as when the user does not give enough details to single out a unique referent (e.g., "what is the price of that t-shirt?" instead of "what is the price of the t-shirt on the left?") @xcite . if a conversational agent needs to answer an ambiguous utterance of the user like the example, the agent may not be able to predict a correct answer effectively. therefore, for an agent to generate a correct answer to the ambiguous utterance of the user, it is critical to find out the information about objects mentioned in the utterance and help the agent to generate the answer. to solve this problem, we use the simmc2.1 dataset. for each scene in the data, there is a json file with every object's information including its name, id, bounding box (bbox), location, and relationship. we use the objects' metadata(e.g., color, pattern, type) to determine whether the current user utterance is ambiguous and extract all objects mentioned in the user utterance predicted as ambiguous. figure 1 is the structure of our proposed model. we construct a two-step pipeline model to solve the task. first, we detect whether the current user utterance is ambiguous using a binary classification model. then, we use the dialogue history and object information to extract all objects in the ambiguous utterance of the user. this paper has two main contributions as follows: experiments show that our method can improve the model performance. as a result, we gained 3rd place in sub-task 1 of the simmc2.1 track. ## method we solve the ambiguous candidate identification task with a two-step pipeline model. figure 1 is the structure of our proposed model. we describe the disambiguate detection in 3.1 and the ambiguous candidate identification in 3.2. ## conclusion our proposed model places 3rd in the ambiguous candidate identification task of the simmc2.1 track. there are two main contributions. first, we consider both the contrastive loss and the binary cross-entropy loss and use a multi-task learning framework, mt-dnn. by learning a model in this way, it can have both benefits of contrastive loss and binary cross-entropy loss. we have demonstrated through testing that our proposed learning method is effective for ambiguous candidate identification. second, by using our prompt to further input metadata information into a model, we create richer embeddings for successful ambigu-ous candidate identification. then, we use a linear combination to adjust the encoding ratio according to the data characteristics to create embeddings suitable for object extraction. through ablation studies, evaluation results show that visual metadata information using prompts improve performance. in particular, model performance increased by about 11%p through visual metadata information (prompt), proving the effectiveness of our prompts. we suggest a novel two-step pipeline system for the ambiguous candidate identification task. in the future, we are planning to work on an end2end system instead of a two-step pipeline system.