‘What are you referring to?’ Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges (2023)

Abstract: Referential ambiguities arise in dialogue when a referring expression does not uniquely identify the intended referent for the addressee. Addressees usually detect such ambiguities immediately and work with the speaker to repair it using meta-communicative, Clarificational Exchanges (CE): a Clarification Request (CR) and a response. Here, we argue that the ability to generate and respond to CRs imposes specific constraints on the architecture and objective functions of multi-modal, visually grounded dialogue models. We use the SIMMC 2.0 dataset to evaluate the ability of different state-of-the-art model architectures to process CEs, with a metric that probes the contextual updates that arise from them in the model. We find that language-based models are able to encode simple multi-modal semantic information and process some CEs, excelling with those related to the dialogue history, whilst multi-modal models can use additional learning objectives to obtain disentangled object representations, which become crucial to handle complex referential ambiguities across modalities overall.

## introduction in dialogue, people work together on a moment by moment basis to achieve shared understanding and coordination @xcite @xcite @xcite . a key mechanism people use to repair misunderstandings when they occur is via metacommunicative, clarificational exchanges (ce): a clarification request (cr) followed by a response (see fig. 1 ). crs are a highly complex phenomenon: they are multi-modal @xcite , highly context-dependent with different forms and interpretations (purver, 2004; purver 1 not to be confused with, but related to clarification ellipsis as used in e.g. @xcite 2 the source code and evaluation experiments are available at https://github.com/jchiyah/what-are-you-referring-to and @xcite , and can occur at different levels of communication on @xcite joint action ladder @xcite . but while the crucial role of generating and responding to crs in dialogue systems has long been recognised @xcite @xcite , crs still remain an understudied phenomenon @xcite , especially in the context of recent successes in multi-modal dialogue modelling @xcite @xcite @xcite @xcite . there is recent work related to identifying when to pose a cr @xcite @xcite , but few evaluate the ability of models to process their responses @xcite . in this paper, we use crs as a testbed for studying and evaluating different neural dialogue model architectures (see also @xcite ). we focus on referential crs occurring at level three of @xcite action ladder: that of understanding. we provide a framework for evaluating how well multi-modal dialogue models are able to exploit referential ces to resolve ambiguous referential descriptions. we use this framework to probe several state-of-the-art models proposed for the simmc 2.0 challenge @xcite trained to resolve situated multi-modal coreferences with ces found in the simmc 2.0 dataset itself. the results indicate that the ability of a model to exploit crs to resolve referential ambiguities depends on the level of granularity of the model's cross-modal representations, i.e. how well information about different object attributes is represented. in particular, we find that the model that includes a training objective designed for predicting object attributes in a multi-task setup performs significantly better than the rest which was not optimised with this objective. this is in line with findings in @xcite who show that having disentangled object representations @xcite allows models to better partition the search space of potential referents; and thereby better exploit effective object attributes in disambiguation. ## dataset we used the simmc 2.0 dataset @xcite , which is a collection of multi-modal taskoriented dialogues, where both the system and the agent are situated in the same virtual environment. the dataset dialogues have a high degree of ambiguity and use rich referring expressions due to the overlap of many similar-looking objects (e.g., 5 red t-shirts in view); dialogues with references to multiple and previously discussed objects (mean 4.5 unique objects referenced per dialogue, sd: 2.4); and changing points of view throughout dialogues with partially observed objects. thus, referential ambiguities in both the visual and conversational contexts are common. furthermore, other common datasets do not contain coordination phenomena exhibited in simmc 2.0 (i.e. guesswhat?! (de vries et al., 2017)) or have a mixture of crs which focuses solely on multi-modal referential ambiguities (e.g., photobook @xcite ). ## conclusion referential ambiguities are common in situated human conversations. we sometimes cannot fully understand or identify a referred object or event, and thus we engage in clarification exchanges to resolve the ambiguity. in this paper, we analyse how several state-of-the-art models treat clarifications in situated multi-modal dialogues using the simmc 2.0 dataset. we classify the types of clarifications by the disambiguating property exploited and then evaluate the models with subsets of the data. we find that language-based models perform well, yet struggle to benefit from clarifications. on the other hand, vision seems to be an important (but not essential) addition for models, which helps processing multi-modal ces. paired with a strong dialogue context, these types of models can perform reasonably well and carry information across turns to better handle clarifications. finally, encoding relations between objects and their locations, and using additional learning objectives to predict attribute slots seems the strongest architecture for models to handle ces. based on these results, to create improved models that can resolve referential ambiguities in situated dialogues, we need holistic object-centric representations that contain information about attributes and properties @xcite , and that can dynamically change to reflect the information exchanges available in the dialogue context.