Pronunciation-Aware Syllable Tokenizer for N epali Automatic Speech Recognition System (2023)

Abstract: The Automatic Speech Recognition (ASR) has come up with significant advancements over the course of several decades, transitioning from a rule-based method to a statistical approach, and ultimately to the use of end-to-end (E2E) frameworks. This phenomenon continues with the progression of machine learning and deep learning methodologies. The E2E approach for ASR has demonstrated predominant success in the case of resourceful languages with larger annotated corpus. However, the accuracy is quite low for low-resourced languages such as Nepali. In this regard, language-specific tools such as tokenizers seem to play a vital role in improving the performance of the E2E model for low-resourced languages like Nepali. In this paper, we propose a pronunciationaware syllable tokenizer for the Nepali language which improves the results of the E2E model. Our experiment confirm that the introduction of the proposed tokenizer yields better performance with the Character Error Rate (CER) 8.09% compared to other language-independent tokenizers.

## introduction automatic speech recognition (asr) systems have been developed using acoustic, language, and lexical models, employing techniques such as as hidden markov mode (hmm), gaussian mixture mode (gmm), statistical, and probabilistic models traditionally. in deep learning based models, recurrent neural network (rnn) is a popular choice for asr to capture temporal dependencies of speech data. one of the major advantages of rnns for asr is that they can be trained end-to-end (e2e), meaning that the entire system is trained from raw audio input to generate text as output without the need of hand-engineered features. in end-to-end (e2e) asr, the neural network directly learns to map the input audio signal to the corresponding text sequence without the need for intermediate representations. this simplifies the asr development process by eliminating the need for hand-crafted features and complex models including acoustic model, language model, lexicon etc. the gated recurrent uni (gru) @xcite and long short-term memory (lstm) @xcite are the variations of rnn. as per @xcite gru is more effective in terms of computation whereas the lstm is better suited for larger datasets. the transcribed text is considered as a label for supervised learning and needs accurate alignment with speech data. in the context of rnn-based classification, the transcribed text is required to convert into the tokens and them into numerical labels. the tokens are smaller constituent segments of the text. however, the substantial disparity in abstraction levels between the audio signal provided as input and the linguistic tokens generated as output poses a significant challenge for a e2e model to acquire the necessary representations. the common way to tokenize is via the use of languageindependent tokenizer such as character-based, byte pair encodin (bpe) tokenizers. the use of the various tokenization techniques in asr are explored by researchers @xcite and @xcite . the character tokenizer is the simplest tokenizer. the bpe @xcite segments the word into the subwords based on the co-occurrence of the characters and used as tokenization @xcite scheme. sentencepiece @xcite is another languageindependent subword tokenizer that can be trained on the raw sentences. beside the language independent tokenizers, a pronunciation -aware syllable tokenizer is used to divide words into smaller units (phonetic units) called syllables. this type of tokenizer is capable of breaking the text into the language-specific phonetic unit which ensures the appropriate phonetic alignment of token with speech signal. this tokenizer can improve the accuracy of asr and text-to-speech (tts) systems by more accurately aligning the segmentation with natural speech patterns and making it easy to classify them using e2e models. the rest of the paper is organized as follows the related works and the dataset used are explained in section 2 and 3 respectively. section 4 explains the working of the pronunciationaware syllable tokenizer and asr model. section 5 presents the conducted experiments and their results. finally, the paper concludes with section 6 where summary of findings, future plans, and potential extensions to the work are explained. ## related works there are various ways of tokenizing the transcribed text. the language independent tokenizers such as character tokenizer, word, and subword tokenizers (bpe, unigram) are commonly used to reduce the complexity of the tokenization. the character based tokenizer is simplest form of the tokenization scheme on natural language processing (nlp) task. the use of different text tokenizers for asr has been studied for various languages. @xcite has experimented with subword unit-based acoustically aligned tokenization techniques. recently, @xcite has explored the use of pronunciationaware unique character encoding techniques for rnn-based asr models. the experiment showed that the pronunciation-aware character encoding perform better. for low-resource languages, @xcite experimented with the reduce and reconstruct approach to minimize the number of prediction errors. the number of features was decreased by replacing them with characters that sound similar. once prediction is made the replaced characters were substituted to their original form. @xcite also studied a different tokenizer called sentencepiece tokenizer and discovered that this type of tokenizer can be helpful for e2e models. papadourakis et al. (2021) also conducted experiments on phonetically induced subwords for the asr system and observed that connectionist temporal classification (ctc) and rnn-t architectures showed a performance increase of 15.21% compared to other methods. @xcite has described the syllable-based system for the japanese language and implemented it as a subword language model (slm) to coordinate the ctc based acoustic model (am) score. using a syllable-based system on the ctc decoding phase improved the performance of the system. similarly @xcite also explored the use of various tokenizers such as character, syllable, and context-dependent phoneme(cdp) in mandarin speech recognition. this research concluded with a better result when a syllablebased unit is applied on the ctc based asr system. lightweight wordpiece model is proposed by @xcite , which takes word and all training vocabulary as an input and return the longest sub-word segmentation of the input word. @xcite used phonetic alignment technique for the machine transliteration task. as per the authors' experiment the use of the phonetic alignment is superior over phrase-based alignment approach and result is improved around 50% on indic languages. anoop and ramakrishnan (2023) studied the modeling of unit for e2e speech recognition task and concluded that the syllable-based units are best choice in context of the indian languages. the most research that has been done on the nepali asr system development have used character based tokenizer @xcite @xcite @xcite . @xcite has used one-gram text tokenizer on proposed rnn and ctc-based asr model. the authors prepared own dataset by extracting text containing 1320 words and recording with three speakers. they achieved character error rate (cer) of 34%. similarly, @xcite and @xcite used the character based tokenizer on convolution neural network (cnn), gru, and ctcbased asr model and ctc-attention-based encoder-decoder model respectively. their reported cer is 11% and 10.3% respectively. dhakal et al. (2022) research on the other possible tokenizers for nepali script is not well explored. the experiment conducted on various tokenization techniques @xcite @xcite @xcite clearly demonstrate that the tokenizer plays vital role in the performance of the asr and that has to be explored on the nepali language as well. beside that, our another motivation for developing the tokenizer is based on the fact that phonetically rich languages like nepali, the conventional tokenizer (word, subword, bpe, unigram or character based) generated tokens do not properly align with the acoustic feature. the proposed syllabic tokenizer split the given text into the syllable which are phonetically aware. the tokenizer can be used for the the language that uses the devanagari script. ## dataset the open slr dataset @xcite is used 1 in this work for comparative study of the proposed and other tokenizer on e2e nepali asr system (explained in section 4.3). this dataset contains raw recordings in f lac encoding and transcribed text in the devanagari script. the metadata of the speech corpus are summarized in the open slr dataset for nepali speech has three major problems-1) background noise 2) large gaps in between the word speech, and 3) not normalized transcribed text. for noise reduction we generated a time-smoothed spectrogram using an adaptive infinite impulse response (adaptive-iir) digital filter @xcite and mask is computed on this spectrogram, then the mask is smoothed with a filter 1 http://openslr.com/54 ## experiment and results the e2e asr model as explained in section 4.3 and depicted in figure 2 is trained with publicly available speech corpus from open slr @xcite . the tokens are produced by tokenizers such as bpe, character, unigram-based tokenizers and our proposed tokenizer. this gave us a solid base for evaluating the proposed tokenizer. for bpe and unigram tokenization we used sentence-piece @xcite library. the token size is summarize in if we compare cer with other independent research such as the one conducted by @xcite , the performance has improved by 2.21%. ## conclusion and future work we conducted an experiment using a syllable tokenizer that incorporates pronunciation awareness to segment nepali text into subword units or syllables. we applied this tokenizer to an e2e asr framework for aligning transcribed text with speech signals. after introducing the tokenizer in the pipeline, we found that the performance of the model outperformed the state-of-the-art character-based tokenizer by 1.21% and 3.77% in terms of cer and wer respectively. this improvement proves that the syllable-based tokenizer which is pronunciation aware is very crucial for phonetically rich and morphologically complex languages like nepali. there are several areas for improvement in this research. we have plans to use this tokenizer to further investigate its role in various e2e-based models and frameworks like aed, t-aed, and rnn-t. similarly, hyperparameter-based estimation can be performed on the existing model to determine the best values for parameters such as kernel size and number of epochs needed. the open slr dataset @xcite for nepali language does not have standard separated training and validation data set. in the next phase, we are planning to augment more data and investigate this dataset and propose the appropriate training and test split.