Task Proficiency-Aware Dialogue Analysis in a Real-Time Cooking Game Environment (2025)

Abstract: Real-time collaborative dialogue tasks require dynamic, instantaneous decision-making and seamless coordination between participants, yet most existing studies on cooperative dialogues primarily focus on turn-based textual environments. This study addresses the critical gap in understanding human-human interaction patterns within dynamic, real-time collaborative scenarios. In this paper, we present a novel dataset collected from a real-time collaborative cooking game environment inspired by the popular game “Overcooked.” Our dataset comprises detailed annotations of participants’ task proficiency levels, game scores, game action logs, and transcribed voice dialogues annotated with dialogue act tags. Participants exhibited a broad range of gaming experience, from highly proficient players to those with minimal exposure to gaming controls. Through comprehensive analysis, we explore how individual differences in task proficiency influence dialogue patterns and collaborative outcomes. Our findings reveal key dialogue acts and adaptive communication strategies crucial for successful real-time collaboration. Furthermore, this study provides valuable insights into designing adaptive dialogue systems capable of dynamically adjusting interaction strategies based on user proficiency, paving the way for more effective human-AI collaborative systems. The dataset introduced in this study is publicly available at: https://github.com/UEC-InabaLab/OverCookedChat.

## introduction human-ai collaboration is emerging as a critical research area within the field of ai @xcite @xcite . moving beyond the traditional scope of ai as mere assistance or automation tools, this paradigm aims to tackle more sophisticated problems by synergistically leveraging the complementary strengths of both humans and ai @xcite . reflecting this trend, there is a surge of research focusing on collaborative frameworks where ai is treated not just as a tool but as a partner, enabling humans and ai to work together effectively towards shared goals @xcite @xcite @xcite . while research exists on human-ai collaboration for task completion through dialogue @xcite , many existing environments primarily feature turn-based cooperation, often neglecting the crucial aspect of realtime interaction. addressing this gap, liu et al. extended the collaborative cooking game environment @xcite , which was modeled after the popular game overcooked, specifically to assess human-ai collaboration in a real-time setting. this environment imposes strict time limits and necessitates high-frequency interactions and rapid responses from both the human player and the ai agent, simulating the pressures of timesensitive teamwork. within this demanding context, they developed an agent possessing the requisite response speed and reasoning abilities to engage in collaborative tasks with humans in this real-time environment @xcite . however, their agent primarily reacts to the human user's actions and explicit instructions, effectively casting the user into a fixed, directive role (i.e., the commander). this imposed asymmetry forces the user to constantly monitor the agent and provide guidance, potentially increasing their cognitive load and hindering the emergence of a truly synergistic and adaptive partnership. such a rigid interaction structure may ultimately limit the effectiveness and fluidity of the collaboration. therefore, to foster natural and seamless collaboration, it is crucial for ai agents to adapt flexibly to their human partners. in this study, we focus specifically on adapting to the user's proficiency with the task. considering user proficiency is vital in environments requiring mutual understanding, such as collaborative work settings. for instance, a reactive dialogue strategy where the agent primarily awaits user instructions might be effective for highly proficient users who prefer to lead. this same strategy could hinder progress and frustrate less experienced users who might benefit from more proactive suggestions or guidance. to investigate effective dialogue strategies tailored to individual user characteristics, particularly proficiency, this study undertakes the collection of human-human collaborative task data. we employ a collaborative cooking game environment previously utilized in @xcite as the platform for this data collection, as shown in figure 1 . the popularity of the original overcooked game also facilitates participant recruitment, particularly for finding individuals with prior experience and potentially higher task proficiency, which is beneficial for our study focusing on proficiency-based adaptation. our primary goal is to analyze this dataset to elucidate the interplay between user proficiency levels, their communication behaviors, and overall task performance, thereby generating actionable knowledge for designing agents capable of adapting to individual users. the main contributions of this work are as follows: (1) we collect and present a novel dataset of human-human collaborative interactions within a real-time, high-pressure task environment. (2) we conduct a detailed analysis of this dataset, identifying dialogue strategies crucial for successful real-time collaboration, with a specific focus on how these strategies vary based on user proficiency levels. (3) we offer actionable findings and design guidelines, grounded in our empirical analysis, to inform the creation of collaborative ai agents that adapt to individual user proficiency. ## design guidelines for collaborative dialogue systems based on the analyses presented in this paper, we propose the following design guidelines for dialogue systems intended to engage in collaborative tasks with human users. first, a fundamental strategy for the system should involve proactive communication regarding its own status and intentions (inform), coupled with frequent expressions of agreement and thanking in response to the user's utterances and actions. our analysis indicated that these dialogue acts are generally associated with higher task performance (as discussed in section 4.2.1). furthermore, given that effective role division was found to be strongly correlated with task success (section 4.1), proactive information sharing is crucial for establishing and maintaining appropriate roles between the user and the system. second, it is critical for the system to dynamically assess or infer the proficiency difference between itself and the human user during the collaborative process and adapt its interaction strategy accordingly. if the system possesses high task proficiency and the perceived proficiency difference with the user is relatively small (low-diff or med-diff scenarios), continuing with the aforementioned strategy of proactive information sharing, agreement, and thanking appears effective (section 4.2.1, 4.2.2 and 4.2.3). however, when the system is highly proficient but perceives a large proficiency gap with the user (high-diff scenario), the system should adopt a more directive role, actively issuing requests or instructions to guide the user and take leadership in the task (section 4.2.4). conversely, if the system has low proficiency while the user is highly proficient (another instance of a high-diff scenario, but with roles reversed), the system should be designed to encourage or solicit guidance and instructions from the user, effectively positioning the user as the leader (section 4.2.4). finally, in situations where both the system and the user have low proficiency (low-diff-l scenario), our findings suggest that communication strategies alone may be insufficient to ensure efficient collaboration (section 4.2.3). in such cases, the primary focus should be on improving the system's fundamental task execution capabilities before sophisticated adaptive communication strategies can become truly effective. ## conclusion this paper presented an analysis of a novel human-human interaction dataset from a real-time collaborative game, investigating how communication patterns and subjective experiences correlate with task success, particularly considering partner proficiency differences. our findings revealed that while some communication acts consistently aid collaboration, the effectiveness of others depends heavily on the partners' relative skill levels. based on these insights, we proposed design guidelines for adaptive collaborative dialogue systems. a cooking game environment figure 6a shows the map used for the initial practice session, while figure 6b depicts the map used for proficiency measurement. both are designed with a simple structure suitable for single-player task execution and feature longer time limits for order completion compared to the collaborative maps, facilitating familiarization and assessment, respectively. figures 6c , 6d , 6e, and 6f show the four collaborative maps used for data collection, which participants played sequentially in this order. figure 6c represents a standard layout for collaborative play, featuring a spatially balanced arrangement of cooking stations. this design allows for relatively easy movement to different stations and facilitates flexible mutual assistance between players based on the evolving situation. figure 6d is designed to encourage clearer role division through its structure and station placement. movement between the left and right areas requires passing through a central passage, promoting a style where each player tends to focus on specific roles while coordinating their efforts. figure 6e features the most rigid role division structure among the maps used, completely separating the two players via a central partition. completing orders necessitates strong interdependence and relies heavily on effective communication for coordinating actions. figure 6f shares a similar structure to figure 6d but introduces greater complexity by incorporating two cooking methods and the maximum of four ingredient types, requiring players to handle more intricate orders. furthermore, while the player on the right side can efficiently perform tasks from retrieving vegetables to chopping, the diversity of orders creates an asymmetric workload distribution. consequently, the player on the left, who typically has a lighter task load, often assumes a crucial role in planning and leading the overall cooking process. figure 7 displays the distribution of game scores for each collaborative map. on the standard map, scores tend to be skewed towards the lower end. this is likely because it was the first collaborative map participants played, making it challenging for them to establish efficient coordination patterns immediately. in contrast, the concave and partition maps show scores distributed across a wider range, suggesting that performance on these maps is more sensitive to the success or failure of the pair's coordination and chosen strategy. the expand concave map also shows a tendency towards lower scores, although less pronounced than the standard map. this is presumably because the map's structure and the complexity of orders hinder efficient collaboration, making it difficult to achieve high scores.