Non-parallel Accent Transfer based on Fine-grained Controllable Accent Modelling (2023)

Abstract: Existing accent transfer works rely on parallel data or speech recognition models. This paper focuses on the practical application of accent transfer and aims to implement accent transfer using non-parallel datasets. The study has encountered the challenge of speech representation disentanglement and modeling accents. In our accent modeling transfer framework, we manage to solve these problems by two proposed methods. First, we learn the suprasegmental information associated with tone to finely model the accents in terms of tone and rhythm. Second, we propose to use mutual information learning to disentangle the accent features and control the accent of the generated speech during the inference time. Experiments show that the proposed framework attains superior performance to the baseline models in terms of accentedness and audio quality.

## introduction the accent transfer task refers to the synthesis of speech with an accent, such as british english for north americans. accent pronunciation is a distinctive form of expression influenced by the native language, the speaker's social group or speaking in a particular region @xcite . in general, people find it easier to talk with others in their own accent group. the use of speech is now widely adopted, for example in the field of chatbots and film dubbing requires research on accent transfer of speech. at present, the accent transfer task for parallel data has achieved sound research results and increasing performance. divided according to training data, the methods are specifically divided into: (1) parallel corpus of different accents of the same speaker using source and target speech content and time alignment @xcite @xcite @xcite . (2) non-parallel corpus of * corresponding author multiple speakers with multiple accents using inconsistent source and target speech content @xcite @xcite @xcite @xcite used a multi-stage trained tts model to achieve transfer of north american accents, australian accents, and british accents, and used a chive-bert pre-training model to enhance the audio effect of accent generation. @xcite added an accent variance adaptor to model the rhythmicity of accent variance, and also enhanced the accent generation audio by using a consistency constraint module. the use of phonetic posteriorgrams (ppg) is an essential idea in the application of non-parallel data @xcite @xcite . @xcite extracted ppg from a chinese pre-trained speech recognition model and then used them in an end-toend speech conversion model based on adversarial learning disentangling. this approach achieved accent transfer from chinese mandarin to tianjin and obtained decent results. however, existing accent transfer works highly rely on a large amount of labelled parallel data or advanced speech recognition models. working with an enormous amount of labeling data is always hectic, labor-intensive, and time-consuming, which is more severe for low-resource languages. this limitation hinders the wider application of accent transfer in low-resource scenarios. hence, it is a timely question: is it feasible to do non-parallel accent transfer task under a unified framework? it is challenging because the speech representation containing various components, including speaker timbre, accent characteristics, and linguistic content, which are difficult to disentangle, especially for non-parallel accent transfer task. the dataset for the task in this paper can be represented as {s following the success of mutual information learning to disentangle speaker information in the one-shot voice conversion (vc) task @xcite , this paper applies the non-parallel databased voice conversion model maskcyclegan-vc @xcite to a more challenging task: voice and accent joint conversion. the source speaker's accent can be converted to the target speaker's accent without changing the linguistic content of the speech. the most challenging task is to achieve effective disentangling of accent features, linguistic features, speaker timbre features and fine-grained embodied modeling of accents in a unified model architecture, and to achieve controlled and effective speaker timbre and accent transfer in the prediction phase of the model. the accurate modeling of phonetic pronunciation tones in the task of accent transfer is crucial. the contributions of this paper are as follows. ## method in this section, we first describe our model architecture. then we introduce the fine-grained accent modeling and adversarial mutual information learning and show how accent transfer between speaker s a and speaker s b on non-parallel datasets with different speakers and accents is achieved. ## conclusions based on the application scenario of accent transfer in non-parallel data sets, this paper proposes a non-parallel accent transfer method based on finegrained controllable accent modeling. it applies a vq-vae network for fine-grained modeling of voice intonation and rhythmic pauses, and then delivers the obtained accent features and speech features to a mutual information-based learning feature disentangler. the features extracted by the trained accent encoder can guide the pitch and rhythm variation of the generated speech in the prediction stage of the converted model, achieving controllable modeling of the various accents. the proposed method generates speech with greatly improved fluency and naturalness, and achieves accent transfer in non-parallel dataset through the application of a unified speech conversion framework.