Data Augmentation for Speech-Based Diacritic Restoration (2024)

Abstract: This paper describes a data augmentation technique for boosting the performance of speech-based diacritic restoration. Our experiments demonstrate the utility of this appraoch, resulting in improved generalization of all models across different test sets. In addition, we describe the first multi-modal diacritic restoration model, utilizing both speech and text as input modalities. This type of model can be used to diacritize speech transcripts. Unlike previous work that relies on an external ASR model, the proposed model is far more compact and efficient. While the multi-modal framework does not surpass the ASR-based model for this task, it offers a promising approach for improving the efficiency of speech-based diacritization, with a potential for improvement using data augmentation and other methods.

## introduction the arabic language is one of the official international languages @xcite ranking as the sixth language in world organizations, the third in the organization of african unity, and the first in the islamic world @xcite . arabic is the native language of more than 400 million people in arabic countries and is spoken by more than 5 million speakers beyond the arab world @xcite . the arabic script consists of 28 basic alphabets, representing consonants and long vowels. as short vowels are not represented in the alphabet or the basic construction of words, diacritics can be employed to indicate them foot_0 .these diacritics are dropped from almost all written text, except documents with pronunciation accuracy that cannot be compromised, such as religious texts or children's books. as a result, arabic text corpora used for building machine learning models as meaning 'land', meaning 'benevolence', and meaning 'wheat' are typically undiacritized. however, studies have shown that including diacritics can lead to substantial improvements in various applications. for instance, @xcite demonstrated that diacritized word embeddings significantly outperform non-diacritized versions. similarly, diacritics enhance the quality of speech datasets used in tasks like text-to-speech (tts) and automatic speech recognition (asr). by incorporating diacritics, these systems can generate more natural-sounding speech, improve pronunciation accuracy, and create more precise transcriptions @xcite . transcribed speech datasets used for building asr models are typically undiacritized. as a result, building diacritized models requires a separate diacritization step, which can be performed on the transcripts using many popular text-based automatic diacritic restoration models like, ali-soft (url, 2023), farasa (qcri, 2020), mishkal @xcite , and camelira @xcite . however, in a recent work, @xcite discussed the degradation in performance of diacritic restoration models when applied to speech datasets, resulting in diacritic error rates ranging between 4-50%. they proposed a diacritic restoration framework that incorporates speech audio as another source of information. in particular, they utilize a pre-trained asr model that is fine-tuned to predict diacritized transcripts as an additional source of input for diacritic restoration. this framework resulted in notable improvements in performance. however, the model relies on an asr model as an extrinsic module, and therefore, its overall performance is constrained by the performance of the asr model. in other words, the model proposed in @xcite is not truly multi-modal. another major limitation in this line of work is the scarcity of diacritized speech corpora that can be used for training both the asr and diacritic restoration models. given the potential advantages of integrating speech features into the model, coupled with the scarcity of such data, we explore a data augmentation strategy to boost the training data volume. in this approach, we generate randomly diacritized inputs and utilize a text-to-speech (tts) model to synthesize speech. we hypothesize that diacritic restoration models overfit to word structure and other textual patterns, so the proposed approach aims to randomize the relationship between diacritics and the underlying text to avoid this kind of overfitting. the proposed data augmentation enables the model to learn from a wide range of diacritic variations regardless of the underlying word structure and textual context to help improve generalization by enforcing the recognition of acoustic over textual patterns. in addition, we propose a multi-modal variant of the model in @xcite , where speech is used directly as input rather than relying on an extrinsic asr model. our experiments show that the proposed data augmentation method improves diacritic recognition performance across all test sets, validating the hypothesis that models typically overfit to the textual context. while the proposed multi-modal architecture improved over the text-only baselines, it still lags behind the asr augmented model described in @xcite . nevertheless, the model offers comparable performance while being more integrated and efficient, demonstrating the first implementation of a multi-modal diacritic restoration model. ## related work speech-based diacritic restoration: the majority of diacritization models primarily operate on text-based approaches, utilizing either rule-based methodologies @xcite @xcite or statistical methods @xcite @xcite . integration of speech data into automatic diacritization processes has been minimally explored in prior research. the most recent relevant work is @xcite , in which they proposed a diacritization framework that incorporates both text and audio to diacritize speech data sets. specifically, they utilized a pre-trained automatic speech recognition (asr) model, which was fine-tuned on diacritized arabic speech data to produce diacritized transcripts of speech utterances. these transcripts were uti-lized as supplementary inputs for diacritic restoration models, consistently enhancing performance compared to text-only approaches. their model's performance is notably influenced by the quality and nature of the training data used to fine-tune the asr model, directly impacting diacritic restoration performance. ## data augmentation for speech-based diacritic restoration building on previous research on data augmentation and multi-modal models in natural language processing, we propose a novel approach that extends these concepts to address the specific challenges of speech-based diacritization. to address the challenge of data scarcity and enhance generalization, we investigate a data augmentation technique to supplement training data by creating a synthetic speech dataset. the main objective of integrating a speech dataset in this augmentation process is to leverage pronunciation features, precisely mapping inputs to the diacritized form, rather than solely relying on textual context for disambiguation. in this context-agnostic approach, our emphasis is on accurately utilizing pronunciation to map letters to their diacritized forms. we augment the datasets through two main steps: (1) randomly generating diacritics for each letter in the text, and (2) converting the diacritized text to audio using a text-to-soeech (tts) model. we applied this data augmentation approach to our training set (discussed in section 5.1), doubling the size of the dataset used for training the diacritic restoration model. ## multi-modal diacritic restoration the architecture for speech-based diacritic restoration proposed in @xcite relies on an external asr model (i.e. whisper @xcite ) used to first process the audio and generate transcriptions. although the model significantly reduced diacritic error rates, its reliance on an external system increases computational demands, in addition to the resources already required for processing asr transcripts in the diacritic restoration model. to create a more efficient model, we propose to directly use the speech as an input to a mutli-modal architecture. we adopt the architecture proposed in @xcite for the text side and the cross-attention mechanism, but we change the speech processing module. more concretely, the architecture (depicted in figure 1 ) consists of a sequence labeling model to process the undiacritized text input and a speech feature extraction model. the two sides are then combined using a cross-attention mechanism, with the query vectors obtained from the text side and the key/value vectors from the speech side. the text sequence labeling model can be any sequence labeling architecture, such as transformer or lstm, as described in @xcite . for the speech feature extraction, we use a pretrained self-supervised model to account for the small size of our training set. our aim is to capitalize on the strengths of both modalities to enhance diacritic restoration accuracy and eliminate the dependency on an extrinsic asr model for a more streamlined and efficient process. we used the self-supervised xls-r model @xcite , a large-scale multi-lingual pre-trained model for speech feature extraction. trained on a vast dataset comprising 436k hours of unlabeled speech, xls-r adopts the wav2vec 2.0 objective @xcite across 128 languages. this objective involves learning contextual representations of speech by predicting masked portions of the audio waveform. the model is typically fine-tuned for downstream tasks such as speech translation, asr, or speech classification. we first fine-tuned the model for asr using our speech training set, which is done by adding a linear layer with ctc loss. the fine-tuning ensures that the model's inner representations are already geared toward the representation of arabic sounds. ## results & discussion in this section, we first examine the performance of the proposed multi-modal diacritic restoration model, and then show the effect of our proposed data augmentation technique. as a baseline, we use the text-only and text+asr models from @xcite as it achieves the best performance reported on the test data (refer to @xcite for comparison with other diacritic restoration models). ## conclusions in this paper, we introduced a novel approach for data augmentation that has consistently improved speech-based diacritic restoration models, resulting in enhanced performance on speech datasets compared to models that rely solely on text, and compared to models that do not use data augmentation. additionally, we explored the potential of a multimodal approach that does not rely on an extrinsic asr system as previously proposed for this task. while the multi-modal approach provides a more efficient and streamlined process, and improves over text-only model when data augmentation is used, it did not reach the performance provided by the asr based model. this work highlights a couple of factors that influence the performance of such models. first, diacritic restoration models are generally limited by the type of data used for training them, and most existing open-source data sets for diacritic restoration are based on classi-cal arabic, leading to poor performance in modern standard arabic. applying data augmentation techniques, such as one proposed in this paper, can lead to better generalization. second, adding speech features where speech data is available results in better performance than text-only models, as the speech contains the low level acoustic features that distinguish these diacritics. finally, while speech-based diacritic restoration works, the performance highly depends on the underlying speech model used to extract speech features. our experiments show that optimizing the performance of the models used for feature extraction is essential, and future work could enhance the performance of the multi-modal architecture by integrating features from models that are first fine-tuned and optimized for asr.