TRACE : Real-Time Multimodal Common Ground Tracking in Situated Collaborative Dialogues (2025)

Abstract: We present TRACE, a novel system for live *common ground* tracking in situated collaborative tasks. With a focus on fast, real-time performance, TRACE tracks the speech, actions, gestures, and visual attention of participants, uses these multimodal inputs to determine the set of task-relevant propositions that have been raised as the dialogue progresses, and tracks the group’s epistemic position and beliefs toward them as the task unfolds. Amid increased interest in AI systems that can mediate collaborations, TRACE represents an important step forward for agents that can engage with multiparty, multimodal discourse.

## introduction when engaging in a shared task, collaborators continually exchange information about goals, obstacles, and next steps, thereby building a shared understanding of the problem, or a "common ground" @xcite . in situations involving hybrid human-ai teams, although there is an increasing desire for ais that act as collaborators with humans, modern ai systems struggle to account for such mental states in their human interlocutors @xcite that might expose shared or conflicting beliefs, and thus predict and explain in-context behavior @xcite . additionally, in realistic scenarios such as collaborative problem solving @xcite , beliefs are communicated not just through language, but through multimodal signals including gestures, tone of voice, and interaction with the physical environment @xcite . since one of the critical capabilities that makes human-human collaboration so successful is the human ability to interpret multiple coordinated modalities in real-time, collaborative ais would need to likewise replicate this ability in live realtime settings, but this remains extraordinarily difficult for machines. our system, trace (transparency in collaborative exchanges) addresses this problem with the following novel and unique contributions in a single system: we demonstrate trace on the task of tracking the common ground that emerges within triads performing a situated collaborative task called the weights task @xcite (fig. 1 ). importantly, our system jointly operationalizes methods previously evaluated in isolation @xcite @xcite , and we do this in realtime while balancing speed and performance. to our knowledge, no previous system has attempted this. trace can be adapted to similar situated collaborative tasks with sufficient data, making it useful for real-time analysis of collaborative problem solving and multimodal communication. we also assess the level of error introduced into multiple features by different levels of live automated processing when compared to manuallyannotated ground truth. trace represents an important advance for for ai systems that can model group collaboration in real-time situated contexts. a video demonstration showcasing multiple aspects of a collaborative interaction is available here. installable code and setup instructions may be found at https://github.com/csu-signal/ trace/releases/tag/naacl-demo , available at present under the mit license. ## related work dialogue state tracking (dst) aims to update the representations of a speaker's (user's) needs at each turn in the dialogue, taking into account past dialogue moves and history @xcite @xcite . dialogue studies provide a technical definition of a "common ground" as a set of shared beliefs among participants in an interaction @xcite @xcite @xcite @xcite . this attribution of mental states to one's interlocutors is central to theory of mind @xcite . such internal states may be communicated not just through language, but nonverbal behavior as well @xcite . understanding nonverbal behavior in multimodal communication has been of longstanding interest in psychology and hci @xcite @xcite , and has recently found increasing relevance to ai systems @xcite @xcite . our work is similar in spirit to the dialogue state tracking challenge (dstc; @xcite ). while both are consistent with clark (1996)'s notion of common ground and may involve a live evaluation, our work is novel in that we address the content of the common ground directly rather than proxies such as goal, and interpret multimodal signals in a situated collaborative task context. similar work that involves situated interaction includes grounding of action descriptions @xcite , and previous work using interactive virtual avatars @xcite @xcite where a common ground can be constructed post hoc (krishnaswamy and pustejovsky, 2020). @xcite , introduced a novel task of common ground tracking (cgt) that automatically identifies the set of shared beliefs and "questions under discussion" (quds) of a group with a shared situated task and goal, using multimodal signals to both extract the propositional content being expressed by task participants @xcite , and their epistemic positionings toward them, to mark which are accepted as facts by the group vs. merely evidenced. @xcite laid out the different modalities that may be used to give an ai system enough information to adequately interpret a collaborative dialogue. trace operationalizes and integrates the aforementioned works in real time. ## system description trace is a modular system that combines features from speech, acoustic, rgb, and depth channels to interpret task participants' linguistic and nonverbal behavior to model their common task-relevant beliefs. descriptions of the individual modules and their relations to previous research are given in sec. 3.1, and fig. 2 shows how they interact. all feature modules specify an output interface or a class representing the data type a module outputs. modules also specify zero or more input interfaces, which they require in order to calculate the output. for example, the propositional extraction module requires only text input while the dense paraphrasing module requires text, gesture, and object inputs (fig. 2 ). trace enables modules to set their input interfaces as dependencies, and the contents of the required output interface will be automatically passed into the dependent input interface. thus, the entire system, or any such system built with trace can be structured as a directed graph with features as vertices and edges connecting a module and all of its dependencies. this framework allows for swapping in and out different multimodal processing modules to create variants of the system. trace is demonstrated on the weights task @xcite , a situated collaborative task where triads work together to determine the weights of five differently-colored blocks using a balance scale. the block weights follow the pattern of the fibonacci sequence in increments of 10 grams. the correct weight assignments by color are: 10g (red), 10g (blue), 20g (green), 30g (purple), and 50g (yellow). beliefs in the weights task constitute evidence for or against weight assignments for blocks, or agreement upon the weight of a given block, as discussed in @xcite . fig. 1 shows the physical task space, with blocks and the balance scale on a table with 3 participants seated around it. the task is recorded using an azure kinect rgbd camera, and either a single mxl ac-404 procon microphone or 3 individual lavalier or headset mics-one for each participant @xcite . the system as presented in the demonstration video runs on an alienware aurora r12 tower with an nvidia rtx 3090 with 24gb of vram but can run on systems as small as a laptop with an rtx 3070 ti (8gb vram). see appendix c for further details. ## evaluation we evaluate over groups 1, 2, 4, and 5 of the weights task dataset (wtd). these groups have been fully manually annotated with ground truth labels for all speech transcriptions, gestures, block locations, epistemic "move" labels, and expressed propositions. we use move and proposition models that exclude the relevant test group from the training data. we also present an evaluation of the demonstration video (linked in sec. 1) where the models used were trained over the entire wtd. following @xcite , our primary metric is @xcite ; @xcite ). this can be computed against each utterance (as in fig. 3 ), or averaged over a dialogue (as in we compare trace's live performance to post hoc results from @xcite , who considered only utterances that were annotated as expressing some epistemic position, and used humanannotated dense paraphrases and gesture annotations using gamr @xcite . we present dsc over all three common ground banks, as well as over the union of fbank and ebank, which approximates the quality of propositional extraction independent of epistemic move classification (because misclassified moves may raise a proposition p to the wrong bank). due to the challenge of real-time processing, our reported numbers are often lower, though we do find a few cases where we match or slightly exceed previous results, such as extracting quds in group 5. generally, live processing does fairly well at tracking the set of quds over time but struggles to assign facts and evidence to the right level. this was also a challenge noted in the original @xcite results. as such, we also compare to results reported in @xcite , who focus on using multimodal dense paraphrasing to identify common ground in the aftermath of human-labeled accep t moves, and hence only report results on fbank. thus, their results can be directly compared to the union of facts and evidence (f ∪ e) in the live condition, as they implicitly assume the contents of other preceding utterances accumulate evidence which is then moved to fact status upon the occurrence of a human-labeled accep t . this represents comparisons to all previouslyreported sota on this task and data, however our numbers represent real-time automated processing of all features considering all utterances (unlike @xcite ), meaning that we are simultaneously detecting and classifying epistemic positioning, and considering all banks of the common ground (unlike @xcite ). like previous results, we see significant variance across groups, indicating the intrinsic challenge of the common ground tracking task. trace overpredicts st at em en t s and underpredicts accep t s in groups 4 and 5, just like @xcite . this leads to propositions correctly being surfaced as evidence but never raised to facts according to the model. we also find that trace approaches or outperforms gpt-4o (as reported in tu et al. ( 2024 )) on fact retrieval in groups 2 and 5 given the assumption of true accep t classification. fig. 3 shows an evaluation over the demonstration video, showing detected common ground vs. the annotated ground truth over time per utterance. this shows a typical pattern in the evolution of common ground as the task unfolds, where the group begins with a full set of quds, over time evidence is surfaced (shown as peaks in ebank), and certain correct facts are agreed upon over time. sequences in the demonstration video can also be explained by individual module performance. the utterance "okay, i guess this one's 10" is correctly paraphrased as "okay, i guess [red] one's figure 3 : dsc of each common ground bank vs. moves in the demo video dialogue. a value may be zero if there is no intersection between the predicted and ground truth sets, but also if the union of the ground truth and predicted sets for that bank is empty, resulting in zero denominator. we treat this case as no similarity. ## conclusion trace addresses the already-challenging problem of common ground tracking in a situated collaborative task, and undertakes the added novel challenge of doing so in real time with live processing of multimodal signals. we integrate epistemic state classification @xcite , propositional extraction @xcite , dense paraphrasing @xcite , and gesture detection (vanderhoeven et al., 2024a), using techniques that appropriately balance speed and performance. trace's dependency graph-based architecture facilitates study of multimodal fusion @xcite , and adaptation to other situations and tasks by easily substituting or adding models and features. for example, modules for posture classification can be introduced to model social dynamics and level of individual task engagement @xcite . additionally, trace's codebase has already been leveraged in ongoing work as a flexible platform that can support multiple different research and demonstration efforts. one such example is @xcite , which uses the underlying trace platform to track nonverbal indicators of group engagement, such as joint visual attention and posture. trace will be of use to researchers in dialogue studies and collaborative problem solving, and can be used in building ai systems that mediate collaboration, such as by inserting probing questions @xcite at key moments. adaptation of real-time common ground tracking with trace to other collaborative task scenarios is straightforward. many modules use off-the-shelf processors like whisper asr, opensmile, and mediapipe @xcite . models for epistemic classification and propositional extraction can be trained on annotated data. propositions for a new task can be deterministically enumerated following @xcite , and the faster but less accurate cosine similarity method requires no new model. our gesture recognition models can be reused as long as participants' are positioned similarly to the weights task. there is a practical limit of ∼5 bodies within the camera fov. future improvements to trace as used in the weights task include also tracking the individual beliefs about the task not shared by the group, moving away from specialized depth cameras through rgb versions of modules like gesture recognition, and improving epistemic move classification through richer representation of modalities like gesture and facial expression. further improvements to and use cases for trace include deploying it in less-constrained, more flexible tasks where conversations may be more ambiguous or more diverse, range in different directions, and cover a wider potential space of propositions. we are currently working on expanding trace's usage into such tasks, such as collaborative construction and annotation of non-verbal indicators in general collaborative settings. additionally, we continue to improve trace's flexibility and codebase organization to allow it to accommodate new models and custom technologies, further permitting researchers to deploy individualized solutions for each modality and scenario of interest.