It Couldn’t Help but Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learning (2024)

Abstract: Active participation in a conversation is key to building common ground, since understanding is jointly tailored by producers and recipients. Overhearers are deprived of the privilege of performing grounding acts and can only conjecture about intended meanings. Still, data generation and annotation, modelling, training and evaluation of NLP dialogue models place reliance on the overhearing paradigm. How much of the underlying grounding processes are thereby forfeited? As we show, there is evidence pointing to the impossibility of properly modelling human meta-communicative acts with data-driven learning models. In this paper, we discuss this issue and provide a preliminary analysis on the variability of human decisions for requesting clarification. Most importantly, we wish to bring this topic back to the community’s table, encouraging discussion on the consequences of having models designed to only “listen in’”.

## overhearers in a conversation as @xcite defined it, in addition to speakers and addressees, 2 a conversation can have sideparticipants, who are part of it but at a given moment are neither of the those two, and overhearers, who are spectators without any rights or responsibilities, e.g. a silent audience or a minute-taker who lacks the opportunity to interfere @xcite . they are further divided into bystanders, if one is aware of their presence, or eavesdroppers, who listen secretly (or at a later time). there is evidence that the very process of understanding differs between addressees and overhearers: while interlocutors actively construct mutual understanding with each other, overhearers only passively consume the product of that process @xcite . speakers can design their utterances while taking different attitudes towards overhearers when they are aware of their presence @xcite , but covert overhearers are not acknowledged at all in the conversation, and can only conjecture about the intended meanings @xcite . although the grounding acts they witness, like backchannels, and the availability of multiple perspectives may indeed aid their comprehension @xcite , the original interaction was opportunistically produced to be understood against the original participants' common ground @xcite . in their corpus analysis of common ground in multi-party interactions, @xcite showed evidence that overhearers reach lower levels of understanding than ratified side participants, who in their turn are not very different from di-2 or producers and recipients. rect addressees, in what they call collective states of understanding. related to that, @xcite showed that observers and participants perceive interactions differently and the experiments by fox @xcite provided evidence that overhearers can more easily comprehend instructions while listening to dialogues than to monologues. clark (1992) even argued that most psycholinguistic subjects are actually overhearers, so theories of language processing may actually be theories of overhearing, due to their lack of interactivity. separating addressees from side participants and accommodating overhearers are salient problems in research on multi-party dialogue @xcite @xcite @xcite . more than a decade ago, @xcite already discussed the limitations of using supervised approaches for learning dialogue strategies. they flagged up three concerns: textual data does not contain the underlying uncertainty measures, instances are treated as local point-wise estimates (instead of the sequences they really are) and exploration of novel strategies is not possible, since the model has access only to the outcomes of the chosen dialogue trajectory originally perpetrated by the humans. this reflects the (offline) overhearering paradigm: a person or agent interpreting a pre-existing conversation and deciding what to do if they were in the original participants' shoes. in nlp, this paradigm is widely used in various modelling steps. let us look closer at four main practices, which may have cascaded effects. ## variability in human grounding acts as humans speak, they can provide positive and negative evidence of mutual understanding @xcite , but modelling their timing and decision-making is challenging. @xcite claimed that "it can be very difficult to efficiently capture regularities in behavioral patterns that lead to similar, but not identical structures". in connection to that, people may take various paths in similar conversational situations @xcite . it is thus an open question how far data-driven supervised learning can get given the inherent variability of explicit (not to mention the latent) collateral signs of grounding. backchannels, a positive evidence of grounding, were demonstrated to involve individual variability, and even idiosyncrasy, possibly due to personality, gender or randomness @xcite . although those works showed some regularity in their timing, the sota for the backchannel prediction task is not very high (.66 weighted f1) @xcite . findings on the variability of human decisions to initiate a cr, a negative sign of grounding, are still sparse. @xcite measured an absolute agreement of 39% among three annotators for scripted dialogues with missing asr information. as another reference, @xcite reported a cohen's κ of 48.45 for clarification in emotional support conversations, which, they claimed, may even be inflated. the task of deciding when to request clarification in collaborative instruction following is under active investigation, but models' performance is still suboptimal @xcite @xcite . recent works on the multimodal codraw dialogue game @xcite argued that this may be due to the variability in human decisions and the limitations of using supervised learning @xcite . ## a brief analysis of regularity in crs in codraw, an instruction follower receives instructions to reconstruct a scene using cliparts (as in figure 1 ). only the instruction giver sees the target scene. @xcite identified all crs (around 11% of the instruction follower's utterances) and defined the task of deciding when to request clarification, where models reached only up to .41 binary f1. what is missing as evidence for the claim that data-driven models cannot fully succeed in learning a "when policy" from human data is the actual human performance on this nlp task, i.e. what overhearers predict. for an initial analysis, we collected a convenience sample with three annotators performing a similar task as the trained models: given a dialogue history and the current state of the reconstructed scene, decide which actions to take and, if needed, request clarification (details in appendix). we randomly selected a sample with 90 instances; in half of them, the original player had produced a cr. the average binary f1 of overhearers with respect to the original decision was .51, not much above what sota models achieve. but the proportion of crs widely ranged from 36 to 85%. among the three annotators, the krippendorff's α was 0.10 and the mean pairwise cohen's κ was 0.18. that is already low, but if we consider the original decision as a fourth annotator, measures are even lower: α was 0.02 and κ was 0.06. this indicates that there was slightly more agreement among overhearers than among addresses and overhearers, but in general there was little agreement on deciding when a cr should be realised. figure 2 presents the main binary decision (whether to request clarification or not) for each of the 90 annotation instances, serving to provide a visual overview of such variability. in terms of surface forms, the average bleu score was 0.11 (@xmath0.10) using the original cr as a source and the produced utterances as a reference. the mean cosine similarity between the embedding of the produced and the original crs was 0.38, 0.29 and 0.36 for the three overhearers. figure 1 shows an example of how diverse the produced clarifications can be, both in form and in content, even when all subjects made the same decision to clarify at a given point. these are preliminary insights from a pilot study. further standardised experiments with a larger sample must be conducted. still, the results are already useful to strengthen the argument that, like backchannelling, human cr decisions lack regularity and overhearers have a much harder task trying to interpret and act upon someone else's grounding acts. decisions depend on how interlocutors distribute grounding costs, as per the principle of least collaborative effort @xcite . besides, there might be adaptive behaviours that models are not capturing @xcite . to continue this investigation, we propose distinguishing between the clarification potential @xcite and the clarification need. the first is a larger set of possibilities for clarification of a given utterance, while the latter refers to the decision of whether and what to clarify taken by a given individual operating with that ut-terance and identifying something worth clarifying. or, in other words, the clarification need, which is in the agent, refers to what was asked among all that could be asked. it is challenging to design experiments that can capture the clarification need among individuals, in particular due to the difficulty in replicating a given dialogue context for different subjects if they are not acting as overhearers. a possible next step is to turn the cr decision into an acceptability task, regarding it as a contrast. for each instance, the annotator would see a set of crs. the actual cr observed in the data should ideally be accepted, but possibly others too. if the original cr falls into the empirical potential, there should be a plausible need for it at that point. such experiment could also aim to measure uncertainty at each turn. ## discussion mutual understanding is crafted by "interacting minds" @xcite . in dialogue, "interlocutors share or synchronise aspects of their private mental states and act together in the world" @xcite . on the other hand, we have shown that the current nlp methodology mostly limits us to learning how overhearers predict discourse representations without the actual joint decision making facet, due to the way that data is produced and annotated, the assumptions behind training mechanisms and the evaluation protocols, each adding a layer of overhearing. what can be a better setup to learn human dialogue behaviour, realising it as a truly interactive process? one needs to move on from one-off supervised learning to sequential models that not only understand dialogues but also participate in them. 3 reinforcement learning provides that framing with a fully accessible and explorable environment @xcite , but somewhat circularly requires a good simulation of an user or interlocutor @xcite @xcite . although llms can serve as speaker simulators, so far they cannot fully model all dialogue phenomena. another possibility are hybrid combinations of supervised and reinforcement learning @xcite , as well as further improvements in techniques like rlhf, 3 see @xcite for a related discussion on the limitations of imitation learning and behaviour cloning for embodied agents. see also @xcite for a discussion on supervised learning and the sequential aspect of an interaction. ppo and dpo. but independently of the learning regime, data-driven approaches, which rely on extracting latent patterns and regularities in a corpus, stumble upon the individual variability of some dialogue phenomena, so that tasks may be ill-defined in datasets. besides, although transcribed dialogue contain clues about the decision making during a conversation, they provide only limited evidence of what participants understood or intended, or their internal states @xcite , which are pertinent for modelling some dialogue decisions and meta-communicative acts. indeed, interfaces do not necessarily have to conform to human behaviour, as long as they can sustain graceful interaction @xcite . but from a cognitive perspective, the current nlp resources do not seem to satisfactorily meet our needs for modelling grounding mechanisms. to study the human mind, do we want cognitive models of how meaning and common ground are constructed or only of how they can be reverse engineered from someone else's interactions? to conclude with this argumentative paper, we wish to encourage more studies on the variability of human grounding acts and its impact in modelling human dialogue strategies. besides, we advocate making the overhearing paradigm explicit whenever it is used in future publications and discussing how it can have influenced reported findings.