Automatic Restoration of Diacritics for Speech Data Sets (2024)

Abstract: Automatic text-based diacritic restoration models generally have high diacritic error rates when applied to speech transcripts as a result of domain and style shifts in spoken language. In this work, we explore the possibility of improving the performance of automatic diacritic restoration when applied to speech data by utilizing parallel spoken utterances. In particular, we use the pre-trained Whisper ASR model fine-tuned on relatively small amounts of diacritized Arabic speech data to produce rough diacritized transcripts for the speech utterances, which we then use as an additional input for diacritic restoration models. The proposed framework consistently improves diacritic restoration performance compared to text-only baselines. Our results highlight the inadequacy of current text-based diacritic restoration models for speech data sets and provide a new baseline for speech-based diacritic restoration.

## introduction the arabic script consists of primary alphabetic characters and secondary diacritics. the alphabetic characters represent consonants and long vowels, while diacritics include short vowels, consonant doubling, and nunation, which is an additional 'n' sound that indicates indefinite nouns in standard and classical arabic. since these diacritics are peripheral rather than main alphabetical characters, proficient arabic language users typically omit them. as a result, most text resources, including speech transcriptions, are non-diacritized, and currently available arabic datasets are heavily under-specified for pronunciation. contextual information provides a basis for implicitly filling out the missing information for proficient speakers, but new learners, nonnative speakers, as well as low-resource speech recognition and synthesis models, often struggle to identify the correct sense and pronunciation due to the lack of diacritics. consequently, many tasks aimed at achieving higher performance necessitate diacritizing the script as a preliminary step. to address these challenges, text-based diacritic restoration models (i.e., models that take undiacritized text as input and produce diacritized output) have been employed to address the issue of missing diacritics for text and speech applications @xcite @xcite @xcite . in speech applications, such as automatic speech recognition (asr) and text-to-speech synthesis (tts), text-based diacritic restoration models have been employed in various ways. some previous works employed automatic text-based diacritizers to restore the diacritics in speech transcriptions and train a diacritized asr system @xcite . alternatively, asr models can be trained without diacritics, and text postprocessing can be employed afterwards to restore the diacritics @xcite . in tts, datasets are curated carefully for phonetic coverage, so the training speech transcriptions are carefully annotated and manually diacritized. for instance, the classical arabic text-to-speech corpus (clartts) @xcite was extracted from a recorded classical arabic audiobook and was manually diacritized and verified to ensure consistency. the reliance on manual diacritization means that all available datasets for tts are relatively small, making tts a low-resource application in arabic. automatic diacritization is often used in deployed tts systems to pre-process the text before synthesis. uni-modal text-based diacritic restoration models may not be optimal for speech applications @xcite . speech utterances are typically less structured than text and may have unusual grammar, repetitions, or missing context, which results in domain and style shifts that lead to poor generalization of these models when applied on speech transcripts. further, in asr applications, the output may contain transcription errors and misspellings that further sabotage the text diacritization models used in post-processing. the existence of paired text and speech data presents an opportunity for incorporating an additional modality for disambiguation and diacritic restoration. as shown in @xcite , asr models trained to directly produce diacritics outperform text-based diacritizers applied on asr outputs by a large margin. given the existence of large speech datasets that contain paired undiacritized texts with speech audios, we explore the potential to improve the performance of automatic diacritic restoration for these datasets. 1 this kind of automatic diacritization could potentially enable the development of large diacritized speech corpora for both asr and tts. in particular, we look into whether the speech signal could facilitate more robust diacritization for speech-based datasets compared to text-only diacritic restoration models. to that end, we propose a diacritic restoration framework that incorporates a pre-trained diacritized speech recognition model. our experiments show that the framework improves performance compared to an equivalent text-only model, which presents a promising direction for speech-based diacritic restoration. our findings can be summarized as follows: ## proposed framework diacritic restoration is a sequence labeling task: the input is a sequence of arabic characters without diacritics, and the output is the target diacritic for each input character, or 'no diacritic' if there are none. in our problem setting, we have a dataset that consists of speech utterances along with their undiacritized transcripts. rather than applying a text-based diacritizer on the text transcripts in isolation, we propose a diacritic restoration framework that incorporates both speech utterances and their text transcripts to produce more accurate diacritics. the proposed model is illustrated in figure 1 . for the speech modality, we utilize a pre-trained asr model to produce provisional diacritized transcripts 7 . the undiacritized text transcript and asr hypothesis are fed into two separate sequence encoders of identical configuration. to fuse information from the text and speech modalities, we apply cross-attention at the final 7 'provisional' because they contain asr errors that distort both consonants and vowels. notation: for the rest of the paper, we will refer to this proposed framework as text+asr; diacritic restoration models that only rely on the undiacritized text transcripts (as is the case for all the existing baselines) will be referred to as text-only. figure 2 : top: basic transformer self attention and prediction regions with sliding window mechanism for inference. bottom: cross-attention region from asr prediction used in each sub-sequence. ## results & analysis ## discussion & conclusion we described a framework for diacritic restoration that incorporates input from speech utterances as well as text to improve the performance of diacritic restoration models applied to speech data. the model consists of a pre-trained asr model that produces provisional diacritized hypotheses, which are incorporated into a sequence labeling model via a cross-attention mechanism. the proposed framework can be applied to any sequence labeling model, and we experimented with transformer and lstm architectures. we also proposed a sliding window inference method that improves the length generalization of the model so it can be applied more robustly to longer sequences. the proposed framework consistently improved diacritization performance compared to an equivalent textonly model, leading to significant reductions in diacritic error rates on our test set compared to all existing diacritic restoration models. we observed particular performance gains on the classical arabic test set which is consistent with the data used for fine-tuning the diacritized asr and the diacritic restoration models. the model outperformed all text-based baselines that are trained with much larger text data sets, achieving a 45% relative reduction in diacritic error rates compared to the best performing baseline. however, given the high variability in speech datasets, additional manuallydiacritized datasets are required to enable broader generalization across arabic variants (such as msa and dialectal arabic) and genres (e.g. newswire or casual speech). our results show that highly accurate diacritization can be obtained using a relatively small diacritized speech data set used for training the asr model and the diacritic restoration models, which can facilitate the production of larger diacritized corpora for speech applications.