Divide and Conquer: Rethinking Ambiguous Candidate Identification in Multimodal Dialogues with Pseudo-Labelling (2024)

Abstract: Ambiguous Candidate Identification(ACI) in multimodal dialogue is the task of identifying all potential objects that a user’s utterance could be referring to in a visual scene, in cases where the reference cannot be uniquely determined. End-to-end models are the dominant approach for this task, but have limited real-world applicability due to unrealistic inference-time assumptions such as requiring predefined catalogues of items. Focusing on a more generalized and realistic ACI setup, we demonstrate that a modular approach, which first emphasizes language-only reasoning over dialogue context before performing vision-language fusion, significantly outperforms end-to-end trained baselines. To mitigate the lack of annotations for training the language-only module (student), we propose a pseudo-labelling strategy with a prompted Large Language Model (LLM) as the teacher.

## introduction in multimodal dialogues (mm-dialogue), ambiguous candidate identification (aci) @xcite aims to detect all the probable objects in a visual scene that are referred to by a given user utterance, where the reference cannot be uniquely identified. aci is crucial for resolving ambiguities in multimodal conversational systems, as humans often generate ambiguous referring expressions due to factors like brevity, context dependence, and unintentional ambiguity. current state-of-the-art aci models @xcite make two key unrealistic assumptions during inference. first, they assume the availability of a predefined catalog of items that may appear in a scene, and that this catalog remains fixed from training to inference. second, they frame aci as a candidate selection problem, where ground-truth bounding boxes for all objects are provided during inference. these assumptions limit the generalizability of these models to handle objects not seen during training, which is crucial for real-world multimodal dialogue systems. to bridge this gap, we reformulate the aci task as a direct coordinate prediction problem, moving away from candidate selection and eliminating the reliance on predefined catalogs. this reformulation aims to improve the applicability of aci models to more realistic and dynamic multimodal dialogue setting. we introduce a novel approach to this more challenging reformulation of the aci task, as illustrated in figure 2 . our method decomposes the aci task into two distinct stages. in the first, dialogue reference extraction (drex), we extract linguistic information on item types, visual attributes, and spatial information related to any object reference made in the last user utterance. it is important to note that while the focus is on the most recent user utterance, the extraction process considers the en- llm (teacher) drex module (student) prompt: instruction + few examples dialogue history current user utterance pseudo-label dialogue history current user utterance target: pseudo-label (teacher) greg module input: pseudo-label (teacher) target: ambiguos reference co-ordinates pseudo label generation dialogue reference extraction generalized reference grounding scene (image) figure 2 : training setting of the proposed modular approach. pseudo-labels generated by prompted llm are used as a target for training the drex module and as an input for training the generalized reference grounding (greg) module. during inference, references extracted by the student model are used tire dialogue history to ensure comprehensive contextual understanding. subsequently, in the second stage, generalized reference grounding (greg), we predict the visual coordinates for these extracted references. modular vs. end-to-end models although endto-end modeling with multimodal fusion has demonstrated significant advancements in various visual-language grounding tasks, including phrase grounding @xcite , referring expression comprehension(rec) @xcite , and open vocabulary object detection @xcite , we argue that a modular approach presents several advantages for the more complex aci task. firstly, decoupling reference extraction from visual grounding promotes explicit text-only reasoning over the dialogue context, which is crucial for the aci task. secondly, the modular approach mitigates the challenges posed by lengthy language contexts in vision-language fusion by presenting the grounding model with only the essential linguistic information. despite the advantages, a key challenge of the modular approach is the lack of annotated data for training separate modules. specifically, the simmc2.1 dataset used in our experiments lacks annotations for drex. to address this, we propose a semi-supervised learning (ssl) setup where pseudo-labels generated by prompting a large language model (llm) serve as training targets. pseudo labeling @xcite is an established method in semi-supervised learning (ssl) (van engelen and hoos, 2020), which aims to generate (pseudo-)labels for unlabeled data to guide the learning process. typically, pseudo-labels are generated by a teacher model trained on limited labeled data. the emergence of llms that can be prompted to generate labels with very few examples has further reduced the labeled data requirement in language modeling tasks @xcite @xcite . however, to the best of our knowledge, this is the first study to investigate the use of llms as pseudolabel generators for a multimodal dialogue task. visual-language grounding generally seeks to identify regions within an image corresponding to a linguistic query. two distinct tasks within this field are rec @xcite and phrase grounding @xcite . rec specifically targets the identification of a single region that optimally corresponds to a given linguistic expression, where phrase grounding typically focuses on grounding zero, one, or many regions matching with simpler noun phrases. recent visual language pre-trained (vlp) models (such as @xcite ; @xcite ; @xcite have shown their capability in both tasks through task-specific fine-tuning. ## methodology this section first outlines our proposed modules for reformulated aci in multimodal dialogues. then we discuss the training and inference procedures. ## results and discussion firstly we compare the results of our modular approach against respective end-to-end trained baselines. results in secondly, we assess the robustness of the student model in comparison to the teacher model. for this experiment, we generated pseudo-labels for the validation and test splits using teacher model. the performance on the aci task, when pseudolabels from the teacher are presented to the greg module, is shown in the bottom part of 1. the results suggest that there is potential for further improvements with a better student model. ## conclusion in multimodal dialogues, identifying ambiguous candidates is critical due to prevalent nondeterministic references. we introduce a modular strategy that simplifies aci into two tasks, each task leveraging existing methodologies from text-only dialogues and visual-language grounding. to address the scarcity of annotations for training the reference extraction module, which emphasizes intra-language reasoning, we employ a pseudo-labelling technique where a prompted llm serves as the teacher. our experiments with a simple auto-regressive language model as student and two distinct grounding techniques confirm the effectiveness of our approach compared to traditional end-to-end training. although our work focuses on aci in multimodal dialogues, the general approach of modularization with llm-based pseudo-labelling can be extended to other complex multimodal tasks with long language context, such as interactive task completion @xcite . broadly speaking, the emergence of llms would provide an opportunity for more explainable modular approaches for tasks requiring substantial intra-language reasoning.