Multimodal Common Ground Annotation for Partial Information Collaborative Problem Solving (2025)

Abstract: This project note describes challenges and procedures undertaken in annotating an audiovisual dataset capturing a multimodal situated collaborative construction task. In the task, all participants begin with different partial information, and must collaborate using speech, gesture, and action to arrive a solution that satisfies all individual pieces of private information. This rich data poses a number of annotation challenges, from small objects in a close space, to the implicit and multimodal fashion in which participants express agreement, disagreement, and beliefs. We discuss the data collection procedure, annotation schemas and tools, and future use cases.

## introduction in collaborative tasks, participants may convey their beliefs, desires, and intentions (bdi) through language, gesture, gaze, and action. these modalities communicate explicit beliefs, disambiguate references, and signal implicit attitudes, enabling participants with different backgrounds or knowledge to build a shared common ground-the set of task-relevant facts and evidence jointly accepted by the group. the edinburgh map task @xcite is a well-known example of multimodal, conversational, collaborative task annotation and has long served as a benchmark for studying dialogue, spatial reference, and grounding. our work builds on this tradition with a more complex, co-situated construction task with multiple instruction givers and integrates gesture, speech, and action while supporting the study of common ground under structural and spatial ambiguity. in this project note, we briefly describe the collection and annotation of a novel collaborative problem solving dataset centered around this task. the data is being annotated with multiple modal channels, and the process implicates a number of inter-figure 1 : left to right: a builder and 3 directors participating in the collaborative construction task with a partially-completed structure on the board. director 1 (second from left) is indicating the position of a block using a combination of language and gesture with the accompanying utterance "coming towards me then it's the red long block." esting challenges toward creating semantic annotations that are interoperable across modalities. the problem of common ground tracking (cgt) has been addressed in previous work such as @xcite ; @xcite ; @xcite ; @xcite ; @xcite ; @xcite , and @xcite . multimodal approaches to common ground tracking include @xcite and @xcite . however, the tasks addressed in these and similar approaches @xcite suffer from a number of drawbacks, including problems with 1) agreement/disagreement: there are few opportunities for disagreement as the task is well-structured with clear solutions at each step; 2) complexity: cognitive and interpretive complexity is low as disagreements typically center questions of single-step procedures or computations; 3) reusability: once a group has completed the task, they know the answer and cannot organically perform the task again. our task has been designed to mitigate these shortcomings to to enable the robust study of common ground tracking in multimodal dialogue. ## task description the task we focus on is a group collaborative construction task structured to satisfy the three conditions enumerated in sec. 1, that we previously identified as being shortcomings in existing tasks used in the study of multimodal cgt. namely, we designed the task to create meaningful disagreements within the group about the right course of action, be sufficiently complex such that there are multiple likely solutions toward the goal, and allow participants to do the task multiple times by creating a novel goal each session. the task is designed for 4 people: 3 directors and 1 builder (see fig. 1 ). each builder receives a different side view of a 3d structure made of large blocks (see fig. 2 ; the directors receive their images on a personal tablet). there are an assortment of blocks on the table before the group, but only the builder (identified in fig. 1 as the only person without a tablet) is allowed to touch the blocks. the directors are not allowed to show their private images to each other or to the builder. the group must then collaborate to instruct the builder to build a single coherent structure that is consistent with the images given to all the directors. dialogue is free form and there are no restrictions on what the participants may say, do, or ask each other, as long as the directors do not touch the blocks or show their private images to anyone else. since there are four sides to the structure but only three images provided, there may be multiple valid solutions. a novel test pattern is generated at the beginning of each session. thus this task satisfies the 3 desiderata listed above by distributing partial information throughout the group, and also simulates a scenario in which a group of people with different background knowledge, expertise, and skills must collaborate to solve a problem. ## data collection data was collected at 2 sites, both universities in the united states. the task takes place on a tabletop and is captured using 3 microsoft kinect azure cameras to capture different angles of the task space. audio is recorded on a single conferencestyle tabletop microphone. the study was approved by university institutional review boards (irbs) and participants received usd 15.00 each. novel test structures were generated for each group, either manually by the researchers, or procedurally using a script written in the unity game engine. test structures consist of blocks arranged in a 3d grid configuration, and screenshots of 3 side views are taken and distributed to the 3 directors. each session consists of two phases. in the first phase, the test structure contains strictly square or rectangular blocks arranged in a 3 × 3 × 3 grid (wdh -see fig. 2 ), with no gaps permitted in the structure. in phase 2, the footprint of the structure is expanded to 4 × 4 × 3, the blocks involved may have curved or angled components, and gaps in the structure are permitted. in total, after removing recordings with technical or procedural errors, 38 usable group recordings were retained. most were 20-40 minutes in length. ## annotation schemas the technical challenges in annotating this data are manifold. speech overlaps, the objects are close together, actions and gestures may have multiple physical manifestations and interpretations. additionally, complete annotation of one modality frequently depends on information from another modality, creating dependencies in the annotation pipeline (fig. 3 ). finally, the sheer amount of data makes purely manual annotation an infeasible task. therefore, for most modalities, we adopt a semiautomated machine annotation with human validation and post-correction strategy. specific challenges and methods for each individual modality are given in their respective sections below. ## conclusion we have outlined the desiderata, processes, and challenges involved in annotating common ground in a co-situated, multimodal, partially observable collaborative problem-solving task. this type of annotation requires integrating multiple commu-nicative channels with converging dependencies and raises a range of technical, design, and interpretive challenges, for which we have described our approaches and techniques. more broadly, annotation of data of this kind presents challenges familiar to the annotation community, and we hope that our experiences can serve as a useful reference point. although a gold-standard annotation set and corresponding inter-annotator agreement (iaa) analysis are not yet available, developing them remains a priority for future work. we plan to obtain human annotations, quantify agreement using standard measures (e.g., cohen's κ, krippendorff's α), and evaluate the computed annotations against this gold standard. the annotation remains ongoing and a fully-annotated dataset will be released at a future date. the task's multi-party, partial information setting represents a novel contribution in the age of llms. the resulting corpus captures the conversational and information dynamics of a collaboration that is not fully transparent to any of the participants, including any ai system observing the interaction. therefore, in the context of llm-driven agents for problem-solving support or human-ai collaboration, our data captures how each participant expresses their implicit "theory of mind" of the other participants' beliefs and goals. the ability to infer such belief states has been shown to be challenging for modern llms @xcite , and this challenge is amplified by how even granular task-relevant propositions may be expressed multimodally in this task, including through speech, gesture, and actions. thus, to finetune or assess llms for this and similar tasks, or to provide a modern llm or vlm with sufficient information to interpret participant behaviors in context, an interoperable annotation scheme that captures the semantic relations across modalities and across time, is required. the efforts described represent a step toward a corpus that would be suitable for fine-tuning, constructing scenarios suitable for assessment of zero-shot prompting, or for benchmarking the recoverability of information in modalities of interest from other modalities.