Audio-Based Linguistic Feature Extraction for Enhancing Multi-lingual and Low-Resource Text-to-Speech (2024)

Abstract: The difficulty of acquiring abundant, high-quality data, especially in multi-lingual contexts, has sparked interest in addressing low-resource scenarios. Moreover, current literature rely on fixed expressions from language IDs, which results in the inadequate learning of language representations, and the failure to generate speech in unseen languages. To address these challenges, we propose a novel method that directly extracts linguistic features from audio input while effectively filtering out miscellaneous acoustic information including speaker-specific attributes like timbre. Subjective and objective evaluations affirm the effectiveness of our approach for multi-lingual text-to-speech, and highlight its superiority in low-resource transfer learning for previously unseen language.

## introduction text-to-speech (tts) models have achieved remarkable advancements in generating human-like speech with a high degree of clarity and naturalness @xcite @xcite . however, these models necessitate an immense volume of high-quality audio data for training to produce such high-quality synthetic speech. this challenge becomes even more pronounced when extending a mono-lingual tts model to multi-lingual setting, as each additional language requires a proportional increase in training data. to mitigate this data-intensive challenge, @xcite ; @xcite have employed transfer learning techniques to synthesize speech for low-resource languages. nevertheless, a substantial amount of source language corpora is still necessary to generate the linguistic knowledge that can be reused for speech synthesis in low-resource languages. the predominant method employed in current multi-lingual tts systems involve the use of unique language ids assigned to each language figure 1: conceptual visualization comparing prior multi-lingual methodologies that utilize language ids for language representations (upper) with the proposed methodology (lower). the architecture after the text encoder follows @xcite , which we omit for brevity. @xcite @xcite . in other words, during both the training and inference phases, the model requires the target language's unique id as additional input to generate speech in that language. however, this approach presents two significant limitations. first, it is incapable of generating speech for unseen languages that the model has not been trained on. to synthesize audio in an unseen language, the tts model must be retrained from scratch with a new language id specifically assigned to the new language, rendering the expansion to additional languages cumbersome and inefficient. second, merely representing each language with its corresponding fixed label fails to capture the diverse and intricate features inherent to each language. as a result, this limits the model's ability to accurately generate linguistic nuances within synthetic speech. to address these challenges, we introduce a novel approach that directly learns linguistic features from audio input instead of relying on fixed expressions from language ids. this innovative method facilitates the extraction and generation of representations with richer linguistic information. furthermore, our approach enables the synthesis of speech in unseen language with minimal finetuning, requiring as little as ten minutes of data from the target language. remarkably, not only does this method generate high-quality speech in unseen languages, but also improves speech synthesis for seen languages. comprehensive evaluations confirm that our method achieves precise pronunciation in both seen and unseen languages, underscoring its versatility and effectiveness. ## results and discussion for evaluation, we employed mean opinion score (mos) and character error rate (cer) as metrics. the mos test was conducted to assess the quality of the synthesized speech. fifteen evaluators recruited from the amazon mechanical turk platform listened to the synthesized audio samples and rated their naturalness on a likert scale from 1 to 5. however, due to the difficulty of recruiting multilingual experts for the mos evaluation, we employed noresqa-mos @xcite , a mos prediction model, for the multilingual evaluation. additionally, cer was measured to evaluate the intelligibility of the synthesized speech. we utilized pretrained speech recognition model foot_1 : ope-nai's whisper @xcite to transcribe the synthesized audio and calculate cer. ## conclusion in this paper, we proposed a method for multilingual tts that directly extracts language embedding from audio rather than using fixed expressions from language ids. we enhanced the pretrained language encoder with a projection layer to better capture language representations and incorporated speaker adversarial training to remove extraneous speaker information from the language embedding. our experimental results demonstrate that this straightforward approach is highly effective for multi-lingual tts. furthermore, we were able to observe significant improvements compared to previous methodology. whereas previous language id approach was unable to generate speech for unseen languages, our proposed approach allows for the generation of high-quality speech for an unseen language by finetuning the model using just 10 minutes of additional data. this promising outcome suggests that our method could significantly enhance the accessibility and quality of tts across multi-lingual and low-resource settings.