Fast Bilingual Grapheme-To-Phoneme Conversion (2022)

Abstract: Autoregressive transformer (ART)-based grapheme-to-phoneme (G2P) models have been proposed for bi/multilingual text-to-speech systems. Although they have achieved great success, they suffer from high inference latency in real-time industrial applications, especially processing long sentence. In this paper, we propose a fast and high-performance bilingual G2P model. For fast and exact decoding, we used a non-autoregressive structured transformer-based architecture and data augmentation for predicting output length. Our model achieved better performance than that of the previous autoregressive model and about 2700% faster inference speed.

## introduction the voice generated by speech synthesis is getting used to through various real-world services, such as dubbing in video contents and news article-reading services. in text-to-speech (tts) system, there are several text processing like text normalization, predicting phonological phrasing and grapheme-tophoneme (g2p). as one of them, the g2p module convert text to phonemes and it has been studied in various ways @xcite @xcite @xcite . monolingual g2p researches have the most conducted, although recently bilingual or multilingual g2p research is also being actively performed @xcite @xcite . most of the proposed models with high performance are based on autoregressive transformers @xcite in both monolingual and multilingual g2p. however, these models suffer from high inference latency, which is sometimes unacceptable for real-time tts applications that generate long speech synthesis sounds, such as news sentences. a previous study @xcite used a simple model structure with a few features and batch inference for fast inference speed; however, there were limitations in specific language characteristics. in this paper, we propose a high-performance bilingual g2p model that has an fast inference speed that enables real-time service. for an efficient expression for each language, byte-level representation input and a language index are used as the main inputs, and for fast decoding, the transformer model is based on a non-autoregressive structured decoder @xcite . because the length of the estimated output used in the non-autoregressive structured decoder has a great impact on the g2p accuracy, a sub-network and a data augmentation technique are used to better infer the output length. in addition, we experimented with the difference between training the whole input unit (sentence) and the tokenized unit. we conducted experiments for different language systems, such as european which have a small number of graphemes and east asian ones which have a large number of graphemes. we chose two languages for bilingual g2p model; english and korean. experimental results showed that, despite significantly losing speed, our non-autoregressive transformer-conditional random field (nart-crf) based g2p model achieved better performance than those of previous art models. when it is applied to an actual service system, we can generate the phonemes of several languages with one model, in addition to the speed and high accuracy applicable to real-time tts application. ## the proposed model this section describes the proposed model for fast bilingual g2p conversion. the overall structure of the model is shown in figure 1 . ## conclusion in this study, a structure of a nart-crf was proposed for fast bilingual g2p with real-time processing. for bilingual, input of byte representation was used, and additional sub-network and data augmentation techniques were used for accurate output length inference. the proposed model showed higher accuracy than the existing art-g2p and at the same time showed about 27 times faster inference speed. in addition, when applied to an industrial tts system, the speed was improved to a level capable of real-time processing. in future work, we will study a model with contextual information or representation of language model to solve some error cases caused by lack of context. furthermore, we will experiments with fast "multilingual" g2p by expanding the language types to chinese, japanese, and european languages. as a result of testing two different language system (i.e. european and east asian), it is expected that expansion of languages, which others in same language group, will be possible. additionally, considering the accents and tones used in languages such as english and chinese, and training on an unbalanced dataset remain issues to be resolved.