Transformer-based N epali Text-to-Speech (2023)

Abstract: Research on Deep learning-based Text-toSpeech (TTS) systems has gained increasing popularity in low-resource languages as this approach is not only computationally robust but also has the capability to produce state-ofthe-art results. However, these approaches are yet to be significantly explored for the Nepali language, primarily because of the lack of adequate size datasets and secondarily because of the relatively sophisticated computing resources they demand. This paper explores the FastPitch acoustic model with HiFi-GAN vocoder for the Nepali language. We trained the acoustic model with two datasets, OpenSLR and a dataset prepared jointly by the Information and Language Processing Research Lab (ILPRL) and the Nepal Association of the Blind (NAB), to be further referred to as the ILPRLNAB dataset. We achieved a Mean Opinion Score (MOS) of 3.70 and 3.40 respectively for the same model with different datasets. The synthesized speech produced by the model was found to be quite natural and of good quality.

## introduction deep learning-based tts has achieved significant popularity and success recently. one of the reasons for the increasing popularity is the limited need for manual feature engineering compared to traditional methods like formant, concatenative, and statistical parametric speech synthesis @xcite . the quality of the tts systems in resourceful languages like english, chinese, hindi, etc. seems to be comparatively much better, where the efforts seem to be focused on making the models computationally efficient as well as producing naturalsounding speech. unfortunately, nepali tts research is still in its early stages, and even the existing results require further improvements. some of the limited nepali tts researches include @xcite @xcite . as far as deep learning-based tts research is concerned, most of the research is conducted on autoregressive models like wavenet @xcite , tacotron 2 @xcite . however, these models are computationally expensive and require large datasets. @xcite . non-autoregressive models, on the other hand, are able to synthesize mel spectrograms in the order of magnitude faster than autoregressive ones. the reason behind this is that in non-autoregressive models, the rnns-based acoustic model has been replaced by a transformerbased one. this apparent advantage of the nonautoregressive model demonstrated by fastspeech @xcite and fastpitch @xcite and the results achieved by @xcite encouraged us also to explore the model for nepali language as well. in this paper, we report our attempt to train the fastpitch model for the nepali language with the two different datasets available to us. we also point to different areas of improvement on the trained model in the future. the paper comprises 7 sections which are structured in the following order: section 1 provides the background of the research. section 2 discusses about the existing tts systems. section 3 explains the datasets, models, and frameworks used. section 4 explains about the evaluation process we adopted. section 5 consists discussion about the findings and comparison between existing nepali tts systems. section 6 provides information on improving the developed tts system. ## related works one of the recent nepali tts @xcite used concatenative speech synthesis and unit selection process to generate a natural-sounding voice. they successfully built a nepali tts that was also used along with the non-visual desktop access (nvda). however, still a few issues of overlaps and echoes were reported in the generated speech, which was suspected to be caused by the potential mislabelings between recorded speech phones and the corresponding transcribed characters. @xcite . more recent research has been conducted with autoregressive models like wavenet @xcite and tacotron 2 @xcite ) @xcite @xcite . @xcite trained wavenet on slr43 @xcite and slr54 @xcite . the model achieved a good result but the generated audio lacks the quality of prosody due to the voice sample from different clusters of people. the output has an uncomfortable accent @xcite . similarly, @xcite trained wavenet on custom dataset. however, the results are poor due to the noisy data and insufficient training epoch. following the trends, @xcite and @xcite trained tacotron 2 with wave-glow @xcite and hifi-gan @xcite respectively. @xcite achieved the highest mos score but the model only generates 1-14 seconds of audio @xcite as per the authors, the naturalness of the generated audio can further be improved with more high-quality training data. similarly, (kumar et al., 2023) rigorously explored various tts systems for 13 indian languages across choices of acoustic models, vocoders, supplementary loss functions, training schedules, and speaker and language variations. out of which fastpitch @xcite and hifi-gan @xcite . ## research methodology in this section, we explain the datasets that we used in training the model. besides, we also provide a detailed overview of a sequence of processes followed and the frameworks adopted as part of developing the fastpitch model for nepali. as shown in figure 1 , the research methodology can be described by four major stages, viz., literature review, dataset preparation, training, and evaluation. the detailed overview of dataset preparation, and training strategy is shown in figure 2 , and figure 3 ## evaluation we evaluate our models using subjective mos on a validation dataset of 20 utterances unseen during training. mos is a widely-used metric in which human listeners rate the quality of voice samples on a scale, typically from 1 (worst) to 5 (best), providing an average score that reflects the perceived quality of synthesized speech. out of 24 volunteers, only 13 of the volunteers rated all the audio. as a result, only the responses of 13 volunteers were used to calculate the mos. the result of the evaluation along with the existing tts models is shown in ## results and discussion we were able to achieve mos scores of 3.70 and 3.40 respectively for the munal and danphe models. danphe was trained using 5.07 hours of data and munal was trained using an additional 2.80 hours of data resulting in a total of 7.87 hours. the generated output samples can be accessed via https: //ishandongol.com.np/transformer-tts/ it took us approximately 1.5 days to train danphe's acoustic model for 2500 epochs with a batch size of 32 on our geforce rtx 3080 ti 12 gb gpu. as per @xcite it took them 6.5 days to train the acoustic model and an additional 4 days models mos architecture dataset ab 3.07 wavenet slr43 (2018) & slr54 ((2018)) abe 2.79 wavenet own a 3.60 tacotron 2 & waveglow slr43 (2018) s 4.04* tacotron 2 & hifi-gan slr43 (2018) & own d (ours) 3.40 ± 0.95 fastpitch & hifi-gan ilprlnab (2018) speaker 1 speaker 2 m (ours) 3.70 ± 0.92 3.51 ± 0.91 fastpitch + hifi-gan slr43 (2018) tts models training strategy vocoder ab trained from scratch trained from scratch abe trained from scratch trained from scratch a fine-tuned english model pre-trained on lj (2017) s fine-tuned english model pre-trained on lj (2017) d (ours) trained from scratch pre-trained on lj (2017) m (ours) fine-tuned d pre-trained on lj (2017) to train the vocoder model on nvidia v100 gpu. but, we cannot overlook the fact that they trained on a larger dataset. the comparison of training strategies, vocoder, and the mos scores of different tts models are shown in ## conclusion the results can be further improved with the help of custom-trained or fine-tuned transliteration models. an alternative solution will be eliminating the need for a transliteration module and training directly on unicode as the transliteration approach sometimes runs into the risk of incorrect transliteration. another area for improvement can be using @xcite to extract the ground truth frequencies like mentioned in @xcite . we notice some hindi and newari accents in the generated audio because of some issues of the transliteration model @xcite and slr43 @xcite dataset respectively. our model performs relatively poorly with short utterances, as the ilprlnab dataset we used to train the model consists of a majority of long utterances. preparing a phonetically balanced dataset with both the long and short utterances included with a proper nepali accent will most likely address the issues mentioned above. in addition, we need to focus on named entity recognition (ner) to identify the non-standard words or semiotic classes like numbers, dates, currencies, etc., and verbalizers to convert text from the written domain to the spoken domain. these issues need to be addressed in the future which is expected to further increase the quality of the synthesized speech.