Whisper– TAD : A General Model for Transcription, Alignment and Diarization of Speech (2024)

Abstract: Currently, there is a lack of a straightforward implementation of diarization-augmented speech transcription (DAST), ie. implementation of transcription, diarization and alignment to the audio within one model. These tasks typically require distinct models, necessitating to stack them together for complete processing. In this study, we advocate for leveraging the advanced capabilities of the Whisper models, which already excels in automatic transcription and partial alignment. Our approach involves fine-tuning the model’s parameters on both transcription and diarization tasks in a SOT-FIFO (Serialized Output Training-First In First Out) manner. This comprehensive framework facilitates the creation of orthographic transcriptions, identification of speakers, and precise alignment, thus enhancing the efficiency of audio processing workflows. While our work represents an initial step towards a unified transcription and diarization framework, the development of such a model demands substantial high-quality data augmentation and computational resources beyond our current scope. Consequently, our focus is narrowed to the English language. Despite these limitations, our method demonstrates promising performance in both transcription and diarization tasks. Comparative analysis between pre-trained models and fine-tuned TAD (Transcription, Alignment, Diarization) versions suggests that incorporating diarization into a Whisper model doesn’t compromise transcription accuracy. Our findings hint that deploying our TAD framework on the largest Whisper model could potentially yield state-of-the-art performance across all mentioned tasks.

## introduction speaker diarization (sd) endeavors to ascertain "who spoke when" @xcite . various methodologies have been employed to annotate audio data for the purpose of identifying speakers within it. conventionally, this task was compartmentalized into distinct sub-modules @xcite , ranging from voice activity detection (vad) to clustering speech segments and assigning speaker labels. however, the optimization of each module in isolation restricted overall optimization. with the advent of deep learning techniques, neural networks have been leveraged to improve the performance of these sub-modules by extracting speaker embedding @xcite , thereby rendering models easier to train, more resilient to speaker variability, and robust under varying acoustic conditions @xcite . a recent breakthrough is the adoption of fully end-to-end neural diarization (eend; @xcite ), wherein all sub-modules are replaced by a single neural network. this promising approach enables the joint optimization of model components, potentially enabling the handling of multi-speaker audio and overlapping speech. initially implemented using bi-directional long short-term memory architectures @xcite , these models swiftly transitioned to self-attention-based networks @xcite . nevertheless, challenges persist, including the model's limited capacity to handle a large number of speakers, the difficulty in achieving online processing, and the tendency for models to overfit the training data distribution @xcite . recent advancements have demonstrated that the concurrent modeling of sd and automatic speech recognition (asr) can enhance the performance of both tasks, as exemplified in various models @xcite . this integration allows sd to use both acoustic and linguistic information, resulting in superior performance compared to models relying solely on acoustic information. furthermore, it enables not only to determine "who spoke when" but also discerning "what" was spoken. as discussed in @xcite , various approaches have been explored, including the introduction of speaker tag roles in transcripts @xcite , map-based joint decoding frameworks @xcite , and the emergence of end-to-end speaker attribution asr (e2e sa-asr, @xcite al. (2020a) ), which facilitates speaker counting, multi-talker asr, and speaker identity determination without limitations on the number of speakers. our aim in this research is to unify the diarization and transcription task in one model. we achieved this by fine-tuning existing whisper models, @xcite , which already transcribe speech with state of the art performance and align the transcription to the audio. our fine-tuning enables the recognition of distinct speakers within the speech audio. by focusing on fine-tuning rather than extensive pre-training, we achieve transferable results even with limited data, making our model applicable to languages with minimal available resources. thus, we introduce whisper-tad (transcribe, align, diarize), an initial version of a versatile model that streamlines the dast pipeline. our article is structured as follows; in section 2 we present our methodology, then in section 3 our experimental setup, in section 4 our experimental results and we finally discuss possible further works in section 5. ## experimental results the results of our fine-tuning task on the different whisper models are illustrated in ## further work this study serves as a proof of concept, with further investigations required to fully evaluate the methodology's feasibility. firstly, a crucial step is transitioning from local e2e processing to global e2e processing. this entails enabling the model to consistently assign the same speaker id to speakers across different audio chunks, rather than assigning new speaker ids for each chunk as done in prior research by @xcite using wav2vec @xcite . various approaches can be explored, such as incorporating a classification head by clustering all speakers across the entire audio, thus necessitating an additional output head for the model. secondly, larger versions of whisper need to be fine-tuned to ascertain the maximum performance achievable using this methodology. thirdly, in order to train larger models for the diarization task, data augmentation is indispensable. leveraging datasets like librispeech @xcite for data augmentation can enhance the training process. additionally, data augmentation can facilitate the fine-tuning of these models for diarization tasks in languages with limited accessible resources. ## conclusion this study introduces whisper-tad, a preliminary investigation into a versatile model designed to integrate transcription, sentence-level alignment, and diarization tasks within a unified pipeline. employ-proceedings of clib 2024 ing a sot fifo method, special tokens are incorporated for speaker identification, enabling recognition of up to 5 speakers per 30-seconds audio chunk. our experiments conducted on the isci and ami corpora yield promising outcomes, suggesting potential applicability across languages with limited resources. notably, our approach achieves competitive performance, even in the absence of data augmentation and without the exploration of larger models. these findings underscore the robustness and effectiveness of whisper-tad, offering valuable insights for future research directions in multi-task audio processing.