MONAH : Multi-Modal Narratives for Humans to analyze conversations (2021)

Abstract: In conversational analyses, humans manually weave multimodal information into the transcripts, which is significantly time-consuming. We introduce a system that automatically expands the verbatim transcripts of video-recorded conversations using multimodal data streams. This system uses a set of preprocessing rules to weave multimodal annotations into the verbatim transcripts and promote interpretability. Our feature engineering contributions are two-fold: firstly, we identify the range of multimodal features relevant to detect rapport-building; secondly, we expand the range of multimodal annotations and show that the expansion leads to statistically significant improvements in detecting rapport-building.

## introduction dyadic human-human dialogs are rich in multimodal information. both the visual and the audio characteristics of how the words are said reveal the emotions and attitudes of the speaker. given the richness of multimodal information, analyzing conversations requires both domain knowledge and time. the discipline of conversational analysis is a mature field. in this discipline, conversations could be manually transcribed using a technical system developed by @xcite , containing information about intonation, lengths of pauses, and gaps. hence, it captures both what was said and how it was said 1 . however, such manual annotations take a great deal of time. individuals must watch the conversations attentively, often replaying the conversations to ensure completeness. automated @xcite transcripts could be generated from video-recordings @xcite . however, the potential issue with jeffersonian annotations is that there are often within-word annotations and symbols which makes it hard to benefit from pre-trained word embeddings. inspired by the jeffersonian annotations, we expand the verbatim transcripts with multimodal annotations such that downstream classification models can easily benefit from pre-trained word embeddings. our paper focuses on the classification task of predicting rapport building in conversations. rapport has been defined as a state experienced in interaction with another with interest, positivity, and balance @xcite . if we can model rapport building in the medical school setting, the volunteer actors can let the system give feedback for the unofficial practice sessions, and therefore students get more practice with feedback. also, the lecturer could study the conversations of the top performers and choose interesting segments to discuss. as student doctors get better in rapport building, when they graduate and practice as doctors, treatments are more effective and long-term @xcite @xcite . outside of the healthcare domain, understanding and extracting the features required to detect rapport-building could help researchers build better conversational systems. our first contribution is the identification of multimodal features that have been found to be associated with rapport building and using them to predict rapport building automatically. our second contribution is to include them into a text-based multimodal narrative system @xcite . why go through text? it is because this is how human experts have been manually analyzing conversations in the linguistics community. our text-based approach has the merit of emulating the way human analysts analyze conversations, and hence supporting better interpretability. we demonstrate that the additions bring statistically significant improvements. this feature-engineering system foot_0 could potentially be used to accomplish a highly attention-demanding task for an analyst. with an automated text-based approach, we aim to contribute towards the research gap of automatic visualizations that support multimodal analysis @xcite . the created multimodal transcript itself is a conversational analysis product, which can be printed out on paper. in this paper, we first introduced the problem domain (section 3). secondly, we motivated the new features (detailed in fig. 1 ) to be extracted (section 4). then, we extracted the features from videos and encoded them as text together with verbatim transcripts (section 4). to evaluate whether the text narratives were useful, we ran experiments that predict rapport-building using texts containing different amounts of multimodal annotations (section 5). finally, we discuss the results and visualize the outputs of the system (section 6). ## related works the automated analysis of conversations has been the subject of considerable interest in recent years. within the domain of doctor-patient communication, @xcite calculated session-level input features, including affective features @xcite . analyses using session-level features have a drawback of not being able to identify specific defining multimodal interactions in the conversation @xcite . therefore, we build upon the works of @xcite in addition to the use of session-level features, we propose using a finer level of talk-turn multimodal text representation as inputs into a hierarchical attention network (han) @xcite . we also build upon our previous work @xcite by broadening the range of multimodal features considered. as for the different methods of multimodal information fusion, @xcite completed an extensive review of the different state-of-the-art multimodal fusion techniques. recent multimodal fusion research (such as icon @xcite , cmn @xcite , mfn @xcite , dia-loguernn @xcite , m3er @xcite ) has focussed on end-to-end approaches. unlike the typical end-to-end approach of representing and fusing multimodal features using numeric vectors, our contribution is an entirely text-based multimodal narrative, thereby improv-ing downstream analysis's interpretability. the approach of this system not only annotates the presence of nonverbal events @xcite , but also the degree of the nonverbal event intensity at both the session-level and talkturn-level. ## data this study uses data from the eqclinic platform @xcite . students in an australian medical school were required to complete at least one medical consultation on the online video conferencing platform eqclinic with a simulated patient who is a human actor trained to act as a patient. each simulated patient was provided with a patient scenario, which mentioned the main symptoms experienced. the study was approved by the human research ethics committee of the university of new south wales (project number hc16048). the primary outcome measurement was the response to the rapport-building question on the student-patient observed communication assessment (soca) form, an adapted version of the calgary-cambridge guide @xcite . simulated patients used the soca form to rate the students' performances after each video consultation. our dataset comprises of 873 sessions, all from distinct students. since we have two recordings per session (one of the student, the second of the simulated patient), the number of recordings analyzed is 1,746. the average length per recording is 928 seconds (@xmath0, amounting to a total of about 450 hours of recordings analyzed. the dataset's size is small relative to the number of multimodal features extracted; therefore, there is a risk of overfitting. we used the youtube platform to obtain the transcript per speaker from the recordings. we chose youtube because we @xcite found that it was the most accurate transcription service (word error rate: 0.28) compared to google cloud (0.34), microsoft azure (0.40), trint (0.44), ibm watson (0.50), when given dyadic video-conferences of an australian medical school. @xcite found that among the four categories of youtube errors (omission, addition, substitution, and word order), substitution recorded the highest amount of errors. specifically, they found that phrase repetitions could be mis-transcribed into non-repetitions. from our experience, (a) repairinitiation techniques such as sound stretches (e.g. "ummmm") @xcite , were either omitted or substituted with "um"; (b) overlapping speech was not a problem because our speakers were physically separated and recorded into separate files. we brought together the two speakers' transcripts into a session-level transcript through wordlevel timings and grouped together words spoken by one speaker until the sequence is interrupted by the other speaker. when the interruption occurs, we deem that the talk-turn of the current speaker has ended, and a new talk-turn by the interrupting speaker has begun. the average number of talkturns per session is 296 (@xmath1, and the average word count per talk-turn is 7.62 (@xmath2.2). at this point, we note that acted dialogues differ from naturally occurring dialogues in a few ways. firstly, naturally occurring dialogues tend to be more vague (phrases like "sort of", "kinda", "or something") due to the shared understanding between the speakers @xcite . secondly, taboo words or expletives that convey emotions (like "shit", "pisssed off", "crap") is likely to be less common in an acted medical setting than naturally occurring conversations. some conversations transform into genuine dialogues where the speakers "shared parts of themselves they did not reveal to everyone and, most importantly, this disclosure was met with acceptance" @xcite . this definition of genuine conversation is similarly aligned to our definition of rapport-building in section 4.1. figure 1 shows a summary of the features extracted. we annotated verbatim transcripts with two different levels of multimodal inputs -annotations at the session-level are labeled coarse, whilst annotations at the talk-turn-level are labeled fine. to facilitate comparisons, all input families belonging to the coarse (fine) level would be annotated with uppercase (lowercase) letters, respectively. in this paper, we refer to the previously existing set of features (with white background) as the "prime" ( ) configuration. families are also abbreviated by their first letter. for example, the coarse p family would consist of only speech rate and delay, whilst the coarse p family would consist of p plus tone. as another example, the coarse d family is the same as the d family because there are no newly added features (in blue). we introduce the framework of our multimodal feature extraction pipeline in figure 2 . ## multimodal features extractions as an overview, we extracted the timestamped verbatim transcripts and used a range of pre-trained models to extract temporal, modality-specific features. we relied on pre-trained models for feature extraction and did not attempt to improve on themdemonstrating the value of using multidisciplinary pre-trained models from natural language processing, computer vision, and speech processing for conversational analysis. effectively, we extracted structured data from unstructured video data (section 4.2). with the structured data and verbatim transcript, we weaved a multimodal narrative using a set of predefined templates (sections 4.3 and 4.4). with the multimodal narrative, we employed deep learning techniques and pre-trained word embeddings to predict the dependent variable (section 5). ## experimental settings there are two main types of inputs -(1) numeric inputs at the session-level, and ( 2 ) coarse and/or fine family child id template verbatim transcript 19 transcript returned from the asr system prosody speech rate 20 the doctor quickly said tone 21 the doctor said angrily delay 22 after two hundred milliseconds 23 a long delay actions laughter 24 the doctor laughed nodding 25 the doctor nodded forward trunk learning 26 the doctor leaned forward smiling 27 the doctor smiled posiface 28 the doctor displayed positive facial expression au05, 17, 20, 25 29 the doctor exhibited lip depressor ## experimental results the results are summarized in ## conclusion in this paper, we build upon a fully text-based feature-engineering system. we motivated the added features with existing literature, and demonstrated the value of the added features through experiments on the eqclinic dataset. this approach emulates how humans have been analyzing conversations with the jefferson (2004) transcription system, and hence is human-interpretable. it is highly modular, thereby allowing practitioners to inject modalities. in this paper, we have used a wide range of modalities, including demographics, actions, prosody, mimicry, actions, and history. the ablation tests showed that the added coarse features significantly improve the performance for both decision tree and han models. future research could (1) investigate whether this feature engineering system is generalizable to wider applications of conversational analysis; (2) conduct user studies to validate the usability and ease of interpretability of the visualization.