A Comparative Analysis of Speaker Diarization Models: Creating a Dataset for G erman Dialectal Speech (2024)

Abstract: Speaker diarization is a critical task in the field of computer science, aiming to assign timestamps and speaker labels to audio segments. The aim of these tests in this Publication is to find a pretrained speaker diarization pipeline capable of distinguishing dialectal speakers from each other and an explorer. To achieve this, three pipelines, namely Pyannote, CLEAVER and NeMo, are tested and compared, across various segmentation and parameterization strategies. The study considers multiple scenarios, such as the impact of threshold values, overlap handling, and minimum duration parameters, on classification accuracy. Additionally, this study aims to create a dataset for German dialect identification (DID) based on the findings from this research.

## introduction speaker diarization (sd) models are essential in various applications, including speech-to-text systems. since existing sd systems may not always meet specific requirements, comparing multiple models is necessary to identify the most suitable one, aiding both our research and guiding other researchers. the annual dihard challenge, focusing on sd for challenging audio files, has been held since 2018 @xcite . however, as demonstrated in this contribution to the challenge @xcite and in an paper, which provides an overview of sd systems @xcite , the commonly used evaluation metric is the one called diarization error rate (der), based on false alarm, missed detection, and speaker confusion. in our case this metric is not applicable due to the nature of our ground truth data. the recordings used are from the rede corpus @xcite , featuring 1-2 elderly male speakers translating sentences into their local dialect and an explorer providing this sentences beforehand in standard german. our manually segmented ground truth data only contains these translated sentences from the dialectal speaker, however the original recordings include additional utterances from the dialectal speaker, between these sentences, that we aim to retain. using der could distort results because the sd model might correctly identify an dialectal speaker during these intervals not captured in the ground truth data. therefore, we bypass der and use a dialectal classification model, comparing its accuracy on our manually segmented ground truth data (the resulting performance is our baseline) with the accuracy from the model on audio files created by the different sd pipelines. with clearly separated speakers, the recordings used for testing the model contain only the desired dialectal speaker and thus the model should perform better, because the explorer does not speak any dialect and thus would interfere with classification. we utilize sd models including pyannote @xcite (v2.1), cleaver foot_0 , and nvidia nemo @xcite , highlighting their strengths, weaknesses, and key features. this comparative analysis provides valuable insights, saving researchers time and effort in selecting the most suitable model. the goal is to establish a dataset for german dialect identification (did), advancing research in this field. ## overview speaker diarization (sd) is the task of assigning timestamps and corresponding speaker labels to an audio track. in general, pipelines designed to accomplish this task consist of four sub-tasks. depending on the categorization and assignment, there can be even more sub-tasks, as seen in works such as @xcite , where all possible sub-tasks are analyzed, or in @xcite where pre-and post-processing also constitute sub-tasks. the first of the four sub-tasks is voice activity detection (vad), which identifies when speech is present and removes non-speech sections from the audio. this is followed by segmentation, also known as speaker change detection (scd), which recognizes speaker transitions and divides the audio into individual speaker segments. next, local speaker embeddings (se) are extracted using a pretrained model. these embeddings are then utilized in a clustering algorithm, where speakers with similar embeddings are grouped together to label the speakers globally. ## experimental setup in this section, we outline our experimental setup and the exploration of various parameters for our study. we analyze a total of 20 different ger-man dialects, classified according to wiesinger @xcite , amounting to 60.5 hours of audio data. our focus narrows down to two specific dialects, contributing a combined 11.75 hours of audio, which are already annotated and clearly segmented, allowing them to serve as ground truth. we establish a baseline by assessing the model's accuracy on these manually segmented dialects. the entire pipeline foot_3 , from the normalization of the audios to obtaining the classification accuracy, is depicted in figure 1 . as a first step, all audio files undergo preprocessing to standardize their format. for this we chose a sample rate of of 16khz driven by the requirements of googles trillsson models (shor and venugopalan, 2022), where the largest model is employed for extracting feature embeddings. to extract feature embeddings, the audio segments resulting from speaker diarization (sd) are concatenated per speaker and then divided into segments of 3 seconds each. these segments are then processed through a small convolutional neural network (cnn). the cnn model architecture comprises three dense layers with leakyrelu activations and dropout layers to mitigate overfitting. for validating and testing the dialect classification model, we randomly selected two speakers from each dialect. since the results vary depending on the chosen speakers, the steps of dividing the data into training, validation, and test sets and running the model are repeated 250 times, selecting new random speakers for each run. we then compute the mean accuracy out of these 250 runs. this number of runs has proven sufficient in previous tests to detect significant differences between experiments. for testing if the differences of the experiments are significant we use the mann-whitney u test @xcite . we examine whether there is a significant difference between the baseline and the models accuracy using the resulting audios from various sd models with their default settings. additionally, we evaluate whether there is a significant difference between runs using the resulting audios with the standard settings of each sd pipeline (called standard pipeline) and runs using the resulting audios with parameter adjustments for the sd pipelines or different segment extraction methods. this is indicated by the p-value in the tables, which always refers to the top row of each table. if the distribution of accuracy for the respec-tive experiment is significantly better than that of the top row, the p-value is bolded. parameter adjustments are explained in the upcoming subsections for each sd pipeline, while extraction methods are tested in the same manner for each sd pipeline. in this context, an extraction method means specifying a threshold in seconds, where only the resulting segments of the sd pipeline longer than this threshold are retained. this threshold helps remove non-contiguous utterances, such as clearing one's throat, if they haven't already been eliminated by the sd pipeline. we then incrementally increase the threshold to assess whether the results improve or worsen. ## results