A comparison study on patient-psychologist voice diarization (2022)

Abstract: Conversations between a clinician and a patient, in natural conditions, are valuable sources of information for medical follow-up. The automatic analysis of these dialogues could help extract new language markers and speed up the clinicians’ reports. Yet, it is not clear which model is the most efficient to detect and identify the speaker turns, especially for individuals with speech disorders. Here, we proposed a split of the data that allows conducting a comparative evaluation of different diarization methods. We designed and trained end-to-end neural network architectures to directly tackle this task from the raw signal and evaluate each approach under the same metric. We also studied the effect of fine-tuning models to find the best performance. Experimental results are reported on naturalistic clinical conversations between Psychologists and Interviewees, at different stages of Huntington’s disease, displaying a large panel of speech disorders. We found out that our best end-to-end model achieved 19.5 % IER on the test set, compared to 23.6% achieved by the finetuning of the X-vector architecture. Finally, we observed that we could extract clinical markers directly from the automatic systems, highlighting the clinical relevance of our methods.

## introduction during the last decades, it became easier to collect large naturalistic corpora of speech data. it is now possible to obtain new realistic measurements of turn-takings and linguistic behaviours @xcite . these measurements can be especially useful during clinical interviews as they augment the current clinical panel of assessments and unlock home-based assessments @xcite . the remote automatic measure of symptoms of patients with neurodegenerative diseases could greatly improve the follow-up of patients and speedup ongoing clinical trials. yet, this methodology relies on the heavy burden of manual annotation to reach the necessary amount needed to draw significant conclusions. it is now indispensable to have robust speech processing pipelines to extract meaningful insights from these long naturalistic datasets @xcite ). huntington's disease represents a unique opportunity to design and test these speech algorithms for neurodegenerative diseases. indeed, individuals with the huntington's disease can exhibit a large spectrum of speech and language symptoms @xcite and it is possible to follow gene carriers even before the official clinical onset of the disease @xcite . the first unavoidable computational tasks to extract speech and linguistic information from medical interviews is the diarization: (1) the detection of speaker-homogeneous portions of voice activity @xcite and (2) the identification of speaker @xcite . speaker turns are clinically informative for diagnostic in huntington's disease @xcite . first, a number of studies are trying to solve this problem directly from the audio signal and linguistic outputs, also referred to as speaker role recognition. they are taking advantage of the specificities (ex: prosody, specific vocabulary, adapted language models) of each role in the different domains: broadcast news programs @xcite , meetings @xcite , medical conversations @xcite , child-centered recordings @xcite ## results and discussions figure 3 shows results in term of ier for the different approaches. both approaches greatly improved over chance. if we consider pipelines solving both segmentation and identification, our best performance is obtained using the speaker role recognition approach with @xmath0.5% while the speaker to 23.6%). we ran an additional ablation experiment ( in previous studies in huntington's disease @xcite , the ratio of silence and statistics on utterances were informative to distinguish between classes of individuals. these speech markers can be extracted directly from the predictions of the speaker role recognition outputs. we computed the ratio of silence and the standard deviation of duration of utterances on the test set of the meta-test set m test . this computation was done both from the ground truth segmentation and the segmentation provided by the speaker role recognition system (figures 4 , 5 . we observed that the automatic system outputs behaved differently as a function of clinical marker. the ratio of silence was better predicted (significant spearman correlation of @xmath1.579, @xmath2.009) than the sd of duration of utterances (non significant spearman correlation of @xmath3.325, @xmath4.175). one potential interpretation of our results is that the difference between the ratio and the standard deviation reveals that our pipeline is great overall to obtain summary statistics of the interview, but its precision at the turntaking level is not sufficient to obtain turn statistics. some bias of the predictive system might not hurt the ier metric but hurt the reliability of some clinical measures. ## conclusion and future work detection and identification of speaker turns are fundamental problems in speech processing, especially in healthcare applications. while works studying these problems in isolation has provided valuable insights, in this work, we showed that speaker role recognition was the most suitable approach for interviewees at different stages of huntington's disease. for future work, we plan to investigate the use of these methods to derive robust biomarkers automatically and compare them to more classic approaches @xcite @xcite .