Multi-Level Attention Aggregation for Language-Agnostic Speaker Replication (2024)

Abstract: This paper explores the task of language-agnostic speaker replication, a novel endeavor that seeks to replicate a speaker’s voice irrespective of the language they are speaking. Towards this end, we introduce a multi-level attention aggregation approach that systematically probes and amplifies various speaker-specific attributes in a hierarchical manner. Through rigorous evaluations across a wide range of scenarios including seen and unseen speakers conversing in seen and unseen lingua, we establish that our proposed model is able to achieve substantial speaker similarity, and is able to generalize to out-of-domain (OOD) cases.

## introduction recent years have witnessed significant advancements in speech synthesis research, with notable contributions from well-established models like tacotron @xcite , fastspeech @xcite and vits @xcite . these models have enabled the generation of natural-sounding speech, which has prompted a notable shift in tts research towards the synthesis of speech in the voices of both seen and unseen speakers in the domain of multispeaker tts. however, despite the considerable body of work in speaker imitation, it has primarily operated within the constraints of either a single target language or a predetermined set of languages. consequently, the concept of generating speech in the voice of any speaker, regardless of the language spoken by that individual, remains largely unexplored -a novel concept we introduce as languageagnostic speaker replication. there are two primary research domains dedicated to synthesizing text into audio with a target speaker's voice: voice cloning and multi-speaker tts. voice cloning entails the alteration of a speaker's voice without changing the text, and typically relies on two audio inputs from the source task text speaker speaker language voice cloning p s + u s multi-speaker tts p + u s + u s cross-lingual tts p + u s s language-agnostic tts p + u s + u s + u and target speakers @xcite . although high speaker fidelity is achieved, this task operates within strict textual constraints as just the speaker's identity is modified. in contrast, multi-speaker tts research bypasses the linguistic constraints of voice cloning as it aims to generate speech for any text using a specific speaker's voice. this task requires two modalities during inference: the text to be spoken, and an reference audio of the target speaker's voice @xcite . while the target text may differ entirely from the spoken contents within the target speaker's audio, it still requires that the language being spoken in the reference audio is identical with that of the target text as in voice cloning. unlike the aforementioned tasks, cross-lingual tts studies extend the capabilities of multi-lingual tts systems by allowing them to generate speech in a specific speaker's voice for text in different languages. yet, these endeavors predominantly revolve around addressing the limitations of training data, where each speaker typically speaks just one language @xcite . thus, the primary objective is to facilitate voice transfer across different languages, making in a sense, a speaker a polyglot. more importantly, cross-lingual tts only adapts the voices of seen speakers to a predefined set of languages present within the training data @xcite . on the other hand, the task of language-agnostic speaker replication goes beyond voice cloning, multi-speaker tts, and cross-lingual tts ( in light of these considerations, we pose the following questions: can we achieve high speaker fidelity even when the reference audio's spoken language differs from the target text, and is not represented in the training data? moreover, can this be done in a zero shot manner? our contributions in pursuit of this novel challenge encompass three aspects: 1) we advance zeroshot multi-speaker tts with language-agnostic speaker imitation, 2) investigate a multi-level attention aggregation approach for enhancing speaker fidelity, and 3) demonstrate the effectiveness of our methodology through comparative analyses with baseline models while ensuring a thorough and diverse evaluation by conducting validations for eleven different languages across eight phylogenetic language branches. ## experiments we leverage the train-clean-100 subset of the lib-ritts dataset @xcite to train the entire tts system in an end-to-end manner, which encompasses 53 hours of english recordings distributed among 247 speakers of nearly balanced gender ratio. audios are sampled at 22050 hz and 16 bits. all computational models were trained on a single rtx a6000 gpu for approximately 20 hours until step 300,000. we use β 1 = 0.9, β 2 = 0.98, and ϵ = 10 -9 hyperparameters for adam optimization. our model has a total of 27,056,339 parameters. ## conclusion in this study, we have formulated the novel task of language-agnostic speaker replication, presented a multi-level attention aggregation approach tailored for this task, and explored the intricate interplay of diverse speaker attributes. we have also effectively addressed the fundamental questions we initially posed: 1) even when the language of the reference audio differs from the target text and is unrepresented in the data, high speaker fidelity is achieved. 2) zero-shot speaker replication for both seen and unseen speakers is possible, demonstrating the robustness and versatility of our approach. we anticipate that our preliminary efforts will serve as a foundation for further developments in this domain. in this work, we made several contributions: 1) we defined the task of language-agnostic speaker replication, 2) proposed a novel multi-level attention aggregation method for this task, and 3) conducted comprehensive evaluations spanning multiple languages. our primary focus was on the language-agnostic speaker extraction aspect within the broader tts framework. consequently, we did not extend our current work to include a multilingual context, which would entail synthesizing text inputs in multiple languages, akin to multilingual tts. thus, we intend to undertake this as part of our future research objectives. given the nature of this work, it is imperative to recognize the potential ramifications of unethical utilization such as identity misappropriation and coercive inducement of actions contrary to an individual's consent. we acknowledge these potential risks, and are fully committed to responsible research and usage of voice replication technology.