Which One Are You Referring To? Multimodal Object Identification in Situated Dialogue (2023)

Abstract: The demand for multimodal dialogue systems has been rising in various domains, emphasizing the importance of interpreting multimodal inputs from conversational and situational contexts. One main challenge in multimodal dialogue understanding is multimodal object identification, which constitutes the ability to identify objects relevant to a multimodal user-system conversation. We explore three methods to tackle this problem and evaluate them on the largest situated dialogue dataset, SIMMC 2.1. Our best method, scene-dialogue alignment, improves the performance by ~20% F1-score compared to the SIMMC 2.1 baselines. We provide analysis and discussion regarding the limitation of our methods and the potential directions for future works.

## introduction recent advancements in multimodal dialogue systems have gained more traction in various domains such as retail, travel, fashion, interior design, and many others. a real-world application of multimodal dialogue systems is situated dialogue, where a dialogue agent shares a co-observed vision or physical space with the user, and is responsible for handling user requests based on the situational context, which are often about the objects in their surroundings. this makes multimodal object identification from a dialogue (i.e., identifying objects that fit a dialogue context) an indispensable skill in multimodal dialogue understanding, built on crossmodal understanding to comprehend the relations between linguistic expressions and visual cues. various methods have been proposed to perform multimodal object identification through different paradigms @xcite @xcite figure 1 : multimodal object identification is the fundamental step required to enable multimodal dialogue systems to understand the object referred to by the user. image is adapted from @xcite . @xcite @xcite . these efforts have established remarkable progress in solving this problem. however, aside from an observed gap between the performance of the existing works and human-level performance in multimodal object identification, prior works also rely on a presumption that the information given by the textual context will only lead to specific (i.e., unambiguous) objects, which does not conform to real-world multimodal conversations where ambiguity exists. therefore, in this work, we explore three different solutions to enable multimodal object identification in the situated dialogue system, i.e., dialoguecontextualized object detection, object-dialogue alignment, and scene-dialogue alignment, without adopting the unambiguity assumption. dialoguecontextualized object detection utilizes the spatial and object understanding capability of a pre-trained object detection model, to generate semantic representation containing both visual cues and the spatial understanding of the object. object-dialogue alignment incorporates the image-text alignment capability of clip @xcite , which has been pre-trained on large image-text corpora to perform multimodal object identification from the given dialogue context. scene-object alignment combines the spatial and object understanding capability of a pre-trained object detection model and a pre-trained textual understanding model to produce better semantic vision-language alignment. our contributions are three-fold: ## methodology in this section, we describe the preliminaries of our work ( §3.1) and extensively elaborate on each of our approaches, i.e., dialogue-contextualized object detection ( §3.2), object-dialogue alignment ( §3.3), and scene-dialogue alignment ( §3.4). figure 2 : the architecture of sitcom-detr. sitcom-detr consists of a scene encoder and a dialogue encoder to extract multimodal content, respectively. the dialogue representation is used to guide the object detector module to judiciously filter out unrelated scene objects. ## discussion based on the results and analysis, we show that the scene-object alignment approach is the best performing approach, achieving ∼55-60% f1-score in the multimodal object identification task of simmc 2.1. we analyze the behavior of the model and conjecture that existing lms have a limitation on understanding discourse. additionally, we show the potential benefit of modeling the long-term dependency of dialogue history to further improve the quality of multimodal object identification task ( §5.2). lastly, we analyze the limitation of the existing image-text contrastive approaches for multimodal object identification and propose an alternative objective to alleviate this limitation ( §5.3). for future work, we aim to focus on the scenedialogue alignment methods to further improve the model performance on the multimodal object identification capability. we note five potential points of improvement that can be further explored to improve the model performance in multimodal object identification: 1) the incorporation of cross-object attention in the modality fusion phase to enable a better relative position understanding between objects, 2) the incorporation of linear attention mechanism to handle the long-term dependency of dialogue history, 3) the exploration on better contrastive objectives for multimodal object identification, 4) the exploration on improving discourse understanding for lms to better handle coreference and sudden object shift, and 5) the synthetic scenedialogue data augmentation through the utilization of other publicly available object detection datasets to handle the in-domain data scarcity problem. ## conclusion in this paper, we explore three methods to tackle multimodal object identification and evaluate them on simmc 2.1. our best method, scene-dialogue alignment, improves the performance by ∼20% f1score compared to the simmc 2.1 baselines. we provide an analysis of incorrect predictions by our best approach and the impact of changing the clip learning objective. we further provide discussion regarding the limitation of our methods and the potential directions for future works.