Exploring the Impact of Modalities on Building Common Ground Using the Collaborative Scene Reconstruction Task (2025)

Abstract: To deepen our understanding of verbal and non-verbal modalities in establishing common ground, this study introduces a novel “collaborative scene reconstruction task.” In this task, pairs of participants, each provided with distinct image sets derived from the same video, work together to reconstruct the sequence of the original video. The level of agreement between the participants on the image order—quantified using Kendall’s rank correlation coefficient—serves as a measure of common ground construction. This approach enables the analysis of how various modalities contribute to the constraction of common ground. A corpus comprising 40 dialogues from 20 participants was collected and analyzed. The findings suggest that specific gestures play a significant role in fostering common ground, offering valuable insights for the development of dialogue systems that leverage multimodal information to enhance user the counstraction of common ground.

## introduction understanding the essence of human communication is a crucial challenge in the fields of artificial intelligence and human-computer interaction (hci). the concept of common ground, proposed by clark, refers to the shared knowledge and beliefs between participants in a dialogue, forming the foundation for smooth communication @xcite . unraveling the process of grounding is not only essential for understanding the mechanisms of deep relational building among humans but also holds significant implications for developing ai agents and robots capable of interacting naturally with humans @xcite . recent research has highlighted the influence of multimodal communication channels and social relationships on grounding @xcite . visual cues, in particular, have been shown to facilitate common ground construction, though the spe-cific elements of visual information that are most effective remain insufficiently clarified. additionally, traditional experimental settings often feature tasks with relative ease, making it challenging to conduct a detailed analysis of failures in grounding @xcite . in this study, we propose a novel collaborative task that enables clearer observation of the influence of physical expression as a visual modality and allows for detailed analysis of both successful and unsuccessful grounding instances. the task emphasizes the role of non-verbal communication, enabling precise analysis of how physical modalities, such as gestures and gaze, influence grounding @xcite . by appropriately adjusting task difficulty, the study aims to observe the dynamics of grounding in more realistic scenarios. ## collaborative scene reordering task this study proposes a new collaborative task, the "collaborative scene reordering task," designed to analyze the impact of modality on the construction of common ground. the task aims to examine how physical modalities during dialogue influence grounding and to provide a detailed analysis of this process. the task is designed to meet the following requirements: this task is expected to offer insights into the role of physical modalities in grounding, enhancing our understanding of their contribution to effective communication. the collaborative scene reordering task involves two participants, each possessing separate pieces of information. the task is divided into a transmission phase and a working phase, which alternate as a single set. in this task, participants share their respective information to reorder a set of images according to the narrative flow of a single story. the transmission phase, where participants exchange information, is clearly separated from the work phase, where they physically reorder the images. this separation allows participants to allocate more cognitive resources to communication, encouraging them to focus on physical expressions and their partner's information during the interaction. the task is specifically designed to encourage nonverbal behaviors during the transmission phase. the use of visually dynamic and motion-rich video material as the basis for the images also supports this objective. ## experiment this section describes the experiment conducted using the task proposed in the previous section, aimed at analyzing the influence of modalities in the common ground construction process. ## analysis of collected corpus this section will describe how the dialogue data collected from the experiment was analyzed to examine the process of building common ground. evaluation was performed based on the common ground construction process using information recorded during each phase. the experimental tool recorded the movement of images and the order of images at each time point. numbers were assigned to the images on the tool according to their chronological order. kendall's rank correlation coefficient is calculated based on whether the image arrangements between participant pairs are consistent or inconsistent with each other. this coefficient ranges from -1 to 1, where 1 indicates a perfect positive correlation, -1 indicates a perfect negative correlation, and 0 indicates no correlation. in this study, the objective is not to evaluate whether the sequence of images follows a chronological order but rather to assess the extent to which a shared foundation is accurately constructed through participant interactions. therefore, the coefficient, which indicates the degree of agreement in image arrangement among work phases 1st 2nd 3rd 4th 5th m 0.12 0.15 0.19 0.26 0.28 sd 0.31 0.37 0.36 0.35 0.36 during transmission, cameras set between participants and behind them recorded facial expressions, gestures, and dialogue content during the experiment. the kendall rank correlation coefficient was calculated for each participant pair's work, and changes in the correlation coefficient over time were recorded to analyze the building common ground process (fig. 7 ). to analyze the construction process of the common ground through repeated work phases, we summarize the statistical information on the coefficients at the end of the five work phases ( by separating the work and transmission phases in the task design, we were able to record the common ground construction process during communication. we analyze gestures that contribute to the construction based on the final values for each session. when sessions with a final similarity of the arrangement order between the pairs of participants exceeding 0.7 were considered successful, there were 3 successful sessions (fig. 8 ). ## clustering the grounding process clustering techniques are used to clarify the typical process of grounding in the collected data. the results of clustering using hierarchical clustering, a method for clustering time-series data, are shown. hierarchical clustering is suitable for certain types of time-series data, particularly when the data is represented as fixed-length vectors with fully aligned time steps across all samples. the results of the clustering are illustrated in figure 9 . given the consistency of similar clusters when increasing the number of clusters, we classified the data into four clusters. the vertical axis represents the kendall rank correlation coefficient calculated for each pair of participants' tasks. the horizontal axis corresponds to the step numbers associated with the beginning and end of the five work phases. to delve deeper into the content of dialogues within each cluster, we sampled several conversations from each classified group and analyzed them in relation to their dialogue content. cluster 1: this pattern shows significant progress in grounding common ground early in the task. in these cases, participants tended to share the overall flow and key features using physical expressions, facilitating the grounding process early on. cluster 2: this pattern exhibits steady, average progress in grounding common ground throughout the task. in these instances, participants often identified distinctive characters or motifs from the scenes, progressively building the common ground. some pairs demonstrated a dynamic where one participant led the direction of the flow, while the other followed, contributing to the grounding process. cluster 3: this pattern indicates a general difficulty in establishing common ground. here, participants struggled to share information and find common elements, leading to unsuccessful grounding attempts. cluster 4: in this pattern, significant progress in grounding common ground occurred towards the end of the task. participants successfully shared information over time, leading to successful grounding by the task's conclusion. in some cases, pairs initially built smaller subgroups by sharing parts of the scene flow, then combined these to establish the overall flow. by classifying the data into clusters based on the grounding process, we identified distinct patterns in how common ground develops, offering insights into the dynamics of successful and unsuccessful grounding scenarios. from the video recordings of phases where significant progress in grounding common ground was observed, the following gestures were noted. these gestures are considered to be strong contributing factors to common ground construction: ## summary and future directions in this study, we proposed a novel experimental task, the "collaborative scene reordering task," to analyze the process of grounding common ground in human communication, with a particular focus on the impact of non-verbal modalities. by separating the transmission and work phases within the task, we were able to observe the effects of physicality more clearly and analyze both the successes and failures in the grounding process. we established a method for quantitatively evaluating the grounding process over time using the kendall rank correlation coefficient. clustering was performed based on the grounding process, allowing us to analyze the tendencies in how grounding progresses. furthermore, the study suggested that specific gestures might strongly contribute to the grounding of common ground, affirming the importance of non-verbal communication. these findings not only deepen our understanding of human communication but also suggest potential applications in designing more natural interactions between humans and agents. however, this study has the following limitations. the sample size was small, with a gender imbalance among participants. the task was designed to observe grounding in specific contexts, and caution is needed when generalizing the findings. longterm effects and cultural factors were not considered. due to technical constraints, some non-verbal behaviors may not have been fully captured. future research should include larger and more diverse samples, cross-cultural validation, and investigation of long-term effects. a more detailed analysis of gestures and dialogue content during the interaction will be conducted. by observing the frequency and timing of gestures during the dialogue and performing a quantitative analysis, the aim is to clarify the factors that influence the construction of a shared foundation. furthermore, the current study focuses solely on the physical expressions in the dialogue, without analyzing the content of the dialogue itself. future analysis will include the relationship between dialogue content and gestures, their impact on the construction process, and the effects of different progression strategies during the task. this study provides new insights into the role of non-verbal communication in grounding and makes significant contributions to the fields of hci, the implementation of smoother dialogue systems, and communication research.