“Any Other Thoughts, Hedgehog?” Linking Deliberation Chains in Collaborative Dialogues (2024)

Abstract: Question-asking in collaborative dialogue has long been established as key to knowledge construction, both in internal and collaborative problem solving. In this work, we examine probing questions in collaborative dialogues: questions that explicitly elicit responses from the speaker’s interlocutors. Specifically, we focus on modeling the causal relations that lead directly from utterances earlier in the dialogue to the emergence of the probing question. We model these relations using a novel graph-based framework of *deliberation chains*, and realize the problem of constructing such chains as a coreference-style clustering problem. Our framework jointly models probing and causal utterances and the links between them, and we evaluate on two challenging collaborative task datasets: the Weights Task and DeliData. Our results demonstrate the effectiveness of our theoretically-grounded approach compared to both baselines and stronger coreference approaches, and establish a standard of performance in this novel task.

## introduction recent breakthroughs in generative ai have raised the possibility of systems that follow and interact with multiparty dialogue. inherent in group dialogues are utterance sequences that deliberate on the same information. modeling these is particularly challenging; while such utterances have a linear order and overlapping information, they may be distantly separated in time and the same information may be expressed very differently. in this paper, we construct deliberation chains in dialogue: turn sequences that surface pieces of evidence or questions under discussion that culminate in a "probing utterance," or explicit elicitation of input that does not introduce new information. we model deliberation chains as probing interventions that are preceded somewhere in the discourse by a number of causal interventions, each of which contribute directly to the eventual emergence of the utterance that serves as the probing intervention. without the causal counterpart(s), the probe would not arise in the discourse (at least not in that specific form or at that specific time). both probe and cause are linked to effective group performance @xcite . tracking them requires an evolving understanding of collaborative dynamics and enables disagreement detection, prompting for deeper insights, or analysis of deliberation's influence on individual learning/understanding @xcite @xcite . our novel approach takes inspiration from discourse coherence theory and joint modeling frameworks traditionally applied to coreference resolution. the ability to link probing interventions to their preceding causes in the dialogue is a critical prerequisite for ai systems to support deliberative/collaborative reasoning, which is of interest in domains like education and workforce development. our novel contributions are: our code may be found at: https://github. com/csu-signal/probingdelibration 2 related work collaborative dynamics andrews-todd and forsyth (2020), oecd (2017), and @xcite have all identified the need for teams to construct shared knowledge to function, often through asking questions. @xcite also points out that collaboration requires teammates to initiate interaction. further, @xcite identified conversational repair as a necessary mechanism in forming common ground for a group. @xcite describes the need for team members to externalize their knowledge. @xcite explores how deliberation may lead to team members changing their minds, which is critical for building group common ground @xcite . ## problem formulation segmented discourse representation theory (sdrt) posits that interpreting an utterance involves supplementing its semantics with pragmatic content based on the demands of discourse coherence @xcite . the relation between utterances and prior content required for a full interpretation gives rise to structures which in collaborative dialogues represent the evolution of information that propels such dialogues towards task-completion @xcite . let us define the relevant structures below: definition 1. based on @xcite , let @xmath0, e 1 , e 2 , λ) be a deliberation graph in a collaborative dialogue, that in turn comprises sets of individual deliberation chains. g is characterized as a weakly-connected, weighted, acyclic graph. here, v represents vertices for probing (p) and causal (c) interventions 1 ; edges e 1 denotes connectivity between vertices; weights e 2 indicate causal influence from c to p, thereby establishing a total order; and λ is a directed path induction function over e 2 and a vertex @xmath1, implicit in the discourse's linear order. we formulate deliberation chain construction as a coreference resolution-style clustering problem @xcite , over a dialogue, d, with n utterances, that the system must cluster into probing interventions and their linked causes, such that each cluster forms a unique deliberation chain. given the elements of a cluster, λ reconstructs the chain by enforcing transitive closure over the within-cluster links given the temporal order inherent in the discourse, under definition 1 above. this formulation motivates our joint modeling approach, which is detailed in sec. 5. in fig. 1 , we provide a detailed example of a deliberation chain from our dataset. the causal interventions (e.g., "you have to at least select either the letter a or card 4.") and probing questions (e.g., "can you explain why?") form a structured sequence, where probing interventions are linked to their causal antecedents. this transitive closure forms the deliberation chain, which reflects how participants navigate the problem-solving process. ## dataset annotation we evaluate intervention clustering on two recent, challenging collaborative dialogue datasets: deli-data and the weights task dataset. ## joint learning of deliberation chains prior works in coreference resolution @xcite typically addressed such assignments using joint-learning frameworks that exhaustively score antecedent "spans" and thereby produce coreference chains. our approach implicitly produces the correct chain since interventions in a dialogue follow a linear order assuming transitivity across links. standard joint-learning frameworks for coreference resolution typically operate at the span-level. for our task, where the entire deliberative utterance forms a distinct discourse unit @xcite , this is an incompatible approach. as such, we propose a joint-learning framework that models the task as a conditional probability distribution p r(p, c, l | d), partitioned into multinomial probabilities, assuming that utterance spans are conditionally independent given the dialogue d. mathematically, ## experiments we evaluated our joint modeling method against 3 similarity baselines and two cross-encoder methods adapted from coreference research. ## results we evaluate against coreference methodology using cluster metrics computed using the coval coreference scorer @xcite , specifically b 3 and conll f 1 metrics, as presented for both datasets in the multimodal nature of the wtd likely makes it more challenging than delidata due to cues that may be missed in even the dense paraphrased language. the use of the windowed approach results in a small performance improvement due to the exclusion of false positive links outside w . the bert-cosine and llama 2 zero-shot baselines perform extremely similarly (returning identical metric values on wtd) and achieve perfect or nearperfect recall. this is likely due to these methods returning a very high proportion of false positive links and transitive closure subsequently clustering (almost) all interventions in a dialogue. ## discussion quantitative analysis fig. 5 shows the count of correct links between interventions assigned by the bidirectional baseline and our (non-windowed) joint model for each cluster size. 10 delidata has much longer chains on average with a wider distribution at every chain-size, and the joint model consistently links more pairs correctly in frequent medium cluster sizes, while at larger cluster sizes joint modeling and the bidirectional baseline are competitive. the joint model may be learning a more global representation of deliberation chains, as from a discourse-coherence perspective midsized chains better reflect the true distribution in collaborative dialogues. in wtd, the distribution of cluster sizes is narrower, while dialogues are much longer. our joint model links interventions more conservatively, but also more correctly, than the bidirectional model. this is most evident in the joint model's ∼45-point increase in b 3 precision compared to the bidirectional baseline. the latter's aggressive linking, while boosting recall for mid-sized clusters, does so at a higher cost to precision. this suggests that when the cluster distribution is skewed (smaller chains, longer dialogues), the joint model is better at avoiding impure clusters. ## conclusion and future work in this paper, we established a novel task of deliberation chain construction in collaborative dialogues. we developed a formal graphical model of deliberation chains grounded in discourse coherence theory, and applied coreference resolution techniques to two challenging datasets. our joint modeling approach emerged as the best model on both datasets, setting a performance standard in this novel task. our joint model predicts the probability of an utterance being probing or causal and uses only prior context-a next logical step is to adapt our method to a live interaction, doing intervention detection, and predicting when a probing utterance will (or should) occur. this would represent a significant step forward for ai systems that can mediate realtime collaboration. the wtd's multimodal aspect represents a rich opportunity to investigate multimodality's role in deliberation. for instance, a gesture or action might itself be a probing or causal intervention, and @xcite provide a compatible multimodal framework in which to pursue this. finally, deliberation chains construction is adaptable to interactions with particular characteristics, like argumentation @xcite , and computational understanding these distinctions will open new horizons in human-ai interaction.