Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarization (2023)

Abstract: Speaker diarization is a classic task in speech processing and is crucial in multi-party scenarios such as meetings and conversations. Current mainstream speaker diarization approaches consider acoustic information only, which result in performance degradation when encountering adverse acoustic environment. In this paper, we propose methods to extract speaker-related information from semantic content in multi-party meetings, which, as we will show, can further benefit speaker diarization. We introduce two sub-tasks, Dialogue Detection and Speaker-Turn Detection, in which we effectively extract speaker information from conversational semantics. We also propose a simple yet effective algorithm to jointly model acoustic and semantic information and obtain speaker-identified texts. Experiments on both AISHELL-4 and AliMeeting datasets show that our method achieves consistent improvements over acoustic-only speaker diarization systems.

## introduction speaker diarization(sd) is the task of answering the question "who speaks when" by partitioning audio into segments with speaker identities. in most application settings, the results of speaker diarization are perceived by readers through the assignment of speaker labels to the corresponding words or sentences transcribed from an automatic speech recognition(asr) system. despite the rich profusion of transcribed texts, mainstream speaker diarization systems consider only acoustic information @xcite @xcite @xcite . traditional sd systems usually consist of the following components: (1) voice activity detection @xcite to filter out non-speech frames. (2) extraction of speaker embeddings from the short audio segments, using popular models such as i-vector @xcite , d-vector @xcite and xvector @xcite . (3) clustering embeddings into several classes using algorithms such as agglomerative hierarchical clustering (ahc) @xcite , spectral clustering(sc) @xcite , and hdbscan @xcite . various speaker embedding model and clustering methods have been explored and proposed in @xcite @xcite @xcite . utilizing only acoustic information has significant limitations. for example, the performance of sd system suffers from obvious degradation in adverse acoustic conditions such as noise, reverberation, and far-field recordings. in addition, we often encounter speakers with similar voice characteristics, which pose serious challenge to clustering them into expected classes. given the abundance of transcribed texts present in meetings and conversations, it is of sufficient interest to explore the possibilities of utilizing semantic information to go beyond the limits of acoustic-only speaker diarization. some previous works tried to use semantic information to classify roles in two-speaker conversations, such as doctor-patient conversation and pilotair traffic controller dialogue @xcite . however, these methods are only suitable for specific two-speaker scenarios where the roles are clearlydefined, such as medical diagnosis, job interviews, and air traffic communications. in this work we focus on open multi-party meeting scenarios where the number of speakers is unknown and the relations between speakers are unspecified. speaker identity information has been proven to be beneficial to many downstream nlp tasks @xcite @xcite . however, these works only consider speaker identities as given ground truth @xcite @xcite , which is downstream nlp applications wav asr asr post-processing audio segments embedding extractor text with spk-id spk-turn detection dialogue detection slp module semantic-acoustic integration semantic information forced alignment text with embedding acoustic speaker diarization module plain text speaker diarization system speech recognition system extract speaker-related information for speaker diarization ## conclusions we propose a novel multi-modal speaker diarization system that utilizes two spoken language processing tasks, dialogue detection and speaker-turn detection, to extract speaker-related information from text. these information are then combined with acoustic information to improve the overall performance of speaker diarization. our experiments demonstrate that incorporating semantic information can effectively address the limitations of single-modal speech.