C lini D ial: A Naturally Occurring Multimodal Dialogue Dataset for Team Reflection in Action During Clinical Operation (2025)

Abstract: In clinical operations, teamwork can be the crucial factor that determines the final outcome. Prior studies have shown that sufficient collaboration is the key factor that determines the outcome of an operation. To understand how the team practices teamwork during the operation, we collected **CliniDial** from simulations of medical operations. **CliniDial** includes the audio data and its transcriptions, the simulated physiology signals of the patient manikins, and how the team operates from two camera angles. We annotate behavior codes following an existing framework to understand the teamwork process for **CliniDial**. We pinpoint three main characteristics of our dataset, including its label imbalances, rich and natural interactions, and multiple modalities, and conduct experiments to test existing LLMs’ capabilities on handling data with these characteristics. Experimental results show that **CliniDial** poses significant challenges to the existing models, inviting future effort on developing methods that can deal with real-world clinical data. We open-source the codebase at https://github.com/MichiganNLP/CliniDial.

## introduction in clinical settings, teamwork is crucial for a successful operation, and effective team collaboration can improve the safety and well-being of the patients @xcite @xcite . failures in teamwork and communication among healthcare providers are a major contributing factor to the estimated 250,000 preventable deaths that occur in the u.s. each year @xcite . breakdowns in areas like leadership, situation awareness, decision-making and communication frequently underlie the many forms of * both senior authors contributed equally to this work. to understand how teamwork unfolds in the operating room, we collected clinidial from simulations of medical operations. we collected the audio data, simulated physical signals from the patient manikins, as well as how the team operates from two camera angles. we then annotated behavior codes based on a team reflection behavior framework @xcite to understand how the team members convey their objectives, strategies, and actions during the operation. we provide initial analysis of our dataset, and lay out potential directions in using our dataset. we hope researchers can leverage our dataset creatively, and propose methods to handle real-world clinical data. in this paper, we pinpoint three main characteristics of clinidial, including its label imbalances, rich and natural interactions, and multiple modalities. corresponding to each feature, we design sets of experiments to investigate existing methods' ability to deal with such data, including the large language models (llms) from gpt families and the open-source llama families. experimental results show that clinidial poses significant challenges to these methods. in addition, we invite input from medical professionals to try to bridge the current nlp fields with the real-world applications they expect (appendix f). in summary, our contributions are two folds: trainee support 2 support 4 more dantrolene. (implementation) more dantrolene. (implementation) should we return -how is she staying? you started on some propoful? (seek: actively inviting input) 15:06 15:08 15:09 15:13 we only include the trainee and the two supports in this example, as they are the only three people speaking during this time frame. our real-world setting distinguishes clinidial from existing datasets in various aspects. first, there are significant label imbalances in the collected data. such label imbalances are less common in conventional nlp datasets where researchers have some levels of control over the data distribution by data filtering or downsampling. however, since our dialogues occur naturally in the operation room, the interlocutors are not tasked to generate dialogues but rather to perform the clinical operation and take care of the "patient" as a team. we do not pose any constraints on how the team communicate, and we observe that the amount of majority class labels significantly outmatches the minority class labels. second, there are rich and natural interactions between the team members. compared to the conventional dialogue benchmarks @xcite which typically contain 30 turns at most, the dialogue in our collected dataset contains 311 turns on average. third, there are rich modalities in the collected data. compared to the conventional nlp datasets with text modality @xcite or the conventional multimodal datasets which focus on vision and text modalities @xcite @xcite , the data we collect includes not only the dialogue, but also the corresponding audio, the operation views from two camera angles, and the physiological signals from the "patient" aligned for each timestamp. ## charactersitic i: imbalanced class distribution here we constrain our study within the text domain for handling label imbalance. ## charactersitic ii: conversational nature as discussed in section 3.3, clinidial involves rich interactions among people where they actively communicate information in the operation process. hence, an ideal model would leverage the context information of the interaction. ## conclusion in this paper, we introduced clinidial, a naturally emerged multimodal dialogue dataset collected from clinical operations. unlike existing benchmarks, clinidial addresses real-world complexities such as imbalanced label distributions, rich team interactions, and multiple data modalities. through studies on three key characteristics of our dataset, we found that the best-performing model achieves a macro f1 score of only 51.09, indicating significant room for improvement. this performance suggests that existing methods struggle on clinidial, particularly in handling imbalanced class distributions, leveraging conversational context, and integrating domain-specific multimodal signals. these limitations highlight the gap between existing nlp methods and the demands of real-world clinical applications. we hope clinidial can bridge the gap between advancements in our community and realworld clinical applications. we encourage future research efforts to develop domain-adaptive nlp techniques, improve multimodal fusion strategies, and eventually address the challenges of real-world applications.