Automatic Speech-Based Checklist for Medical Simulations (2021)

Abstract: Medical simulators provide a controlled environment for training and assessing clinical skills. However, as an assessment platform, it requires the presence of an experienced examiner to provide performance feedback, commonly preformed using a task specific checklist. This makes the assessment process inefficient and expensive. Furthermore, this evaluation method does not provide medical practitioners the opportunity for independent training. Ideally, the process of filling the checklist should be done by a fully-aware objective system, capable of recognizing and monitoring the clinical performance. To this end, we have developed an autonomous and a fully automatic speech-based checklist system, capable of objectively identifying and validating anesthesia residents’ actions in a simulation environment. Based on the analyzed results, our system is capable of recognizing most of the tasks in the checklist: F1 score of 0.77 for all of the tasks, and F1 score of 0.79 for the verbal tasks. Developing an audio-based system will improve the experience of a wide range of simulation platforms. Furthermore, in the future, this approach may be implemented in the operation room and emergency room. This could facilitate the development of automatic assistive technologies for these domains.

## introduction in recent years, there is a growing interest in developing performance-based assessment for medical practitioners. in the pursuit for methods that may assess hands-on skills, simulation-based assessment has emerged @xcite . simulation-based assessment requires appropriate validation metrics, and checklists are one of the most common methods. for a given simulation scenario, evaluation experts determine which actions, based on the presenting complaint, are important for the candidate to perform in order to properly manage the scenario @xcite . based on this process, a detailed checklist is developed @xcite @xcite @xcite . during the simulation, an experienced examiner is required for filling in the checklist. the need for an experienced examiner makes the assessment process very time consuming and expensive, and in addition, does not provide medical practitioners the opportunity for independent training. ideally, to reduce the costs of performance assessments and to allow more residents to train in a complex scenario, the process of filling the checklist should be done by a machine: a fullyaware objective system capable of recognizing and monitoring the resident performance. to this end, we have developed a end-to-end fully automatic speech-based objective checklist validation system, capable of identifying anesthesia residents' actions in a simulation environment, based solely on the participants' speech recordings. we developed a simulation setup for data collecting. the checklist system was evaluated using two different clinical scenarios for assessing skills of senior anesthesia residents. our underlying assumption is that in many cases the communication among medical staff may represent the physical action itself. by analyzing the participants' speech, our system can automatically identify and fill the appropriate rubrics in the checklist. ## results as mentioned in section 2.4, a proper syntactically analysis of hebrew texts requires the disassembling of the input tokens down to their constituent morphemes. to evaluate the impact of the lexical analysis on the algorithm results, we compared two versions of the pipeline -one used the lexical analysis and the other didn't. during the process of analyzing the data, we found that few tasks in the checklist tend to be non-verbal in their nature. most candidate don't use any verbal commands when preforming those tasks, and the human observer can validate them only by identifying the action itself. these tasks include 'verification of intubation tube location', 'exposure of patient chest' and few others. these ## discussion in this study we developed a system for automatically filling a medical simulation checklist using audio data. the system is completely autonomous and a fully automatic pipeline from the raw audio files to a complete checklist was established. the system was assessed using novel data collected for this study. the native language of the current participants of this study is hebrew. this poses a unique challenge common to morphologically rich language. as clearly evident from the results, using lexical analysis improved our system performances, and might have a greater impact on a more complex models. we plan to expand our work to other languages in the future and assess the system performance. the system was successful in correctly identifying most of the tasks performed by the participants. yet, one limitation of the system is that it is currently based on keyword matching and not on a more complex model of the conversation. the method in use has limited accuracy, and in addition, only provides a binary score indication whether the task was preformed or not. for example, the cur-rent system may indicate a drug was provided but it will not assess the dosage. in order to develop a more complex algorithm, a significantly larger data base is required. we are continuously collecting data that focuses both on a larger number of participants as well as a wide range of clinical scenarios. this will expedite the development of more complex algorithms. developing an audio-based system has several advantages. first, it may fit to a wide range of simulation platforms including low-and high-fidelity mannequins, virtual reality, and standardized patients. furthermore, in the future, our system could migrate from the simulation domain and be implemented in the operation room and emergency room. this could facilitate the development of automatic assistive systems for these domains.