Speaker Overlap-aware Neural Diarization for Multi-party Meeting Analysis (2022)

Abstract: Recently, hybrid systems of clustering and neural diarization models have been successfully applied in multi-party meeting analysis. However, current models always treat overlapped speaker diarization as a multi-label classification problem, where speaker dependency and overlaps are not well considered. To overcome the disadvantages, we reformulate overlapped speaker diarization task as a single-label prediction problem via the proposed power set encoding (PSE). Through this formulation, speaker dependency and overlaps can be explicitly modeled. To fully leverage this formulation, we further propose the speaker overlap-aware neural diarization (SOND) model, which consists of a context-independent (CI) scorer to model global speaker discriminability, a context-dependent scorer (CD) to model local discriminability, and a speaker combining network (SCN) to combine and reassign speaker activities. Experimental results show that using the proposed formulation can outperform the state-of-the-art methods based on target speaker voice activity detection, and the performance can be further improved with SOND, resulting in a 6.30% relative diarization error reduction.

## introduction speaker identity is an import information for speaker-attributed automatic speech recognition @xcite and dialogue comprehension @xcite , especially in multi-party meeting scenarios @xcite . recently, speaker diarization is developed to detect and track speakers with acoustic features. in general, speaker diarization methods can be divided into three groups, i.e., clustering-based algorithms, end-to-end models and hybrid systems. clustering-based methods comprise three individual parts, including speech segmentation, embedding extraction, and clustering algorithm. specifically, a long-term audio is first split into several segments by voice activity detection (vad). then, speaker embeddings @xcite @xcite are extracted for each segment, which are partitioned into clusters by unsupervised clustering algorithms, such as k-means @xcite , spectral clustering @xcite , agglomerative hierarchical clustering (ahc) @xcite , and leiden community detection @xcite . however, such clustering algorithms are in an unsupervised manner, which does not minimize the diarization errors directly. to overcome this issue, neural models are introduced, such as @xcite and @xcite . recently, the variational bayesian hidden markov model (vbx) is introduced to refine the clustering results @xcite . although vbx achieves impressive performance on several datasets @xcite , it has trouble handling overlapping speech due to speaker-homogeneous assumption. end-to-end (e2e) approaches treat speaker diarization as a multi-label classification problem, where a deep neural network is employed to predict a set of binary labels (speaker activities) for each timestep. in this way, e2e models can deal with overlapping speech and minimize diarization errors directly. @xcite , the utterance-level permutation-invariant training (upit) loss @xcite is employed to train the end-toend neural diarization (eend) model which is further improved in @xcite and @xcite . to deal with an unknown number of speakers, the encoder-decoder based attractor is involved into eend @xcite . another e2e model is the recurrent selective attention network (rsan), which jointly performs source separation, speaker counting, and diarization. although a lot of efforts have been made @xcite @xcite , e2e mod-els still have a low scalability to a large number of speakers and long-term audios due to the label permutation problem and memory limitation. to deal with long-term audios in multi-party meeting scenarios, a hybrid system of clustering and neural diarization model can be a reasonable approach @xcite @xcite . @xcite , the unconstrained k-means algorithm is used to extract x-vectors @xcite as speaker profiles. then, the profiles are consumed by a neural diarization model, target-speaker voice activity detection (tsvad), to obtain frame-level diarization results. tsvad model takes acoustic features along with x-vectors for each speaker as inputs. a set of binary classification output layers produces activities of each speaker. recently, tsvad is widely-used in multi-party meeting scenarios, and achieves the state-of-the-art performance @xcite @xcite . in both tsvad and e2e approaches, overlapping speaker diarization is formulated as a multilabel classification problem. this formulation ignores speaker dependency, and speaker overlaps are not explicitly modeled, which can cause much miss detection and false alarm errors. to overcome the disadvantages, we reformulate overlapping speaker diarization as a single-label prediction problem via power set encoding. in this way, speaker dependency and overlaps are explicitly modeled by predicting a single label of speaker combinations rather than a set of multiply binary labels. to fully leverage this formulation, we further propose the speaker overlap-aware neural diarization (sond) model. in sond, a context-independent (ci) scorer is employed to model global speaker discriminability, while a context-dependent (cd) scorer is involved to discover local discriminability of speakers in the same audio. given ci and cd scores, a speaker combining network is proposed to combine and reassign speaker activities. as a results, the proposed method achieves a 6.30% relative improvement than the state-of-the-art methods on a challenging real meeting scenarios 1 . ## system description an overview of our speaker diarization system is shown in figure 1 . given a long-term audio, we first remove its silence regions according to the 1 our code is available at https://github.com/zhihaodu/ du2022sond results of voice activity detection. then, we uniformly clip the voiced signal into segments with a fixed length. for each segment, a speaker embedding is extracted from a pre-trained neural network. next, we perform spectral clustering on the extracted speaker embeddings, and the clustering centroids are treated as speaker profiles. given the clipped segments and speaker profiles as inputs, a neural diarization model is employed to estimate speaker activities across all timesteps. finally, we perform post-processing on the diarization results of segments and obtain the transcription of entire long-term audio. ## speaker overlap-aware neural diarization in the proposed speaker overlap-aware neural diarization model (sond), input acoustic features and speaker profiles are first projected into the same space by the speech and speaker encoders, respectively. then, the encoded features and profiles are fed into context-dependent and contextindependent scorers. finally, scores of different speakers are combined to predict the power set encoded labels through a speaker combining network. a diagram of sond is shown in figure 2 . ## analysis 6.1 ablation study we conduct the ablation and replacement study to evaluate each component of sond. results are shown in ## conclusion and future work in this paper, a hybrid diarization system improved by pse and sond is proposed for long-term audios in multi-party meeting scenarios. specifically, we perform spectral clustering on the neural network based affinity matrix to extract speaker profiles. then, the profiles are consumed by sond to predict pse labels at different time-steps. we find that explicitly modeling speaker dependency and overlaps via pse labels much reduces diarization errors. in sond, the local speaker discriminabil-ity, discovered by cd scorer, is important for final diarization performance. meanwhile, the global speaker discriminability, modeled by ci scorer, can much improve the robustness to noises in speaker profiles. compared with other network architectures, scn is more efficient for combining and reassigning speaker activities. as a result, our system outperforms the state-of-the-art monaural speaker diarization methods with a 6.30% relative improvement. in the future, we are interested on extending our sond model to leverage spatial information for multi-channel data.