CATT : Character-based A rabic Tashkeel Transformer (2024)

Abstract: Tashkeel, or Arabic Text Diacritization (ATD), greatly enhances the comprehension of Arabic text by removing ambiguity and minimizing the risk of misinterpretations caused by its absence.It plays a crucial role in improving Arabic text processing, particularly in applications such as text-to-speech and machine translation.This paper introduces a new approach to training ATD models.First, we finetuned two transformers, encoder-only and encoder-decoder, that were initialized from a pretrained character-based BERT.Then, we applied the Noisy-Student approach to boost the performance of the best model.We evaluated our models alongside 11 commercial and open-source models using two manually labeled benchmark datasets: WikiNews and our CATT dataset.Our findings show that our top model surpasses all evaluated models by relative Diacritic Error Rates (DERs) of 30.83% and 35.21% on WikiNews and CATT, respectively, achieving state-of-the-art in ATD.In addition, we show that our model outperforms GPT-4-turbo on CATT dataset by a relative DER of 9.36%.We open-source our CATT models and benchmark dataset for the research community .

## introduction the arabic language is characterized by its rich morphology and complex syntactic structure. one of the unique features of arabic is the use of diacritics or tashkeel, which are small marks above or below the letters that indicate vowels or other phonetic aspects of pronunciation. these diacritics are favorable for understanding the meaning of words, as their absence can lead to ambiguities and misinterpretations. they are also crucial in improving performance of applications such as text-to-speech and machine translation @xcite . the diacritics can be affected by the context of the sentence as shown in the following example: in the first sentence, " " or "saqa" is interpreted as "drove", denoting an action. however, when the context changes, the same word, now pronounced as "saqu" takes on a completely different meaning, becoming a noun that translates to "leg". this change highlights the crucial role of diacritics in quickly clarifying the meanings of sentences, as the characters remain the same. at the same time, the pronunciation varies depending on the context. in the previous example, the meaning of the word " " can be comprehended even without diacritics, as long as the reader considers the complete sentence and understands the context provided by the surrounding words. this fact raises the following question: "will a pretrained bert model help in improving the atd models?". in this paper, we propose a training strategy based on a pretrained character-based bert @xcite , and a self-training approach called noisy-student (ns) @xcite . throughout the paper, we will answer the following research questions: ## related work previous research has investigated a broad spectrum of approaches to address the diacritization task, beginning with rule-based methods, moving to classical machine learning models, and reaching sophisticated deep learning architectures @xcite . in addition, comprehensive experiments show that deep learning methods outperform nonneural techniques, particularly when substantial training data is available @xcite . @xcite tested a refined version of the tashkeela dataset @xcite using the shakkala foot_1 model. they also trained a character-level rnn with a block-normalized gradient (bng) module. the bng technique normalizes gradients within each batch, potentially speeding up training and improving generalization @xcite . abbad and xiong's (2020) atd approach consisted of a three-part pipeline: a multi-layer lstm and dense layers, a character-level rule-based corrector for specific error correction, and a word-level statistical corrector that leveraged context and distance information to resolve diacritization issues. furthermore, they developed an enhanced version of the system and named it multilevel diacritizer @xcite . madhfar and qamar (2020) implemented 3 different atd models. the first one was a baseline model consisting of 3 deep bidirectional long short-term memory (bilstm) layers. the second model was an encoder-decoder with 3 lstm layers for the encoder and 2 lstm layers for the decoder. the last model was based on tacotron encoder @xcite that uses cbhg module @xcite . alkhamissi et al. (2020) proposed two architectures: the two-level diacritizer (d2) and the two-level diacritizer with decoder (d3). d3 builds upon the capabilities of d2 by accepting partially diacritized text as input. these models have a wordlevel encoder as well as a character-level encoder. the results of both encoders are combined by an attention mechanism and fed to a unidirectional lstm layer to predict diacritics. @xcite created two deep neural networks (dnns); the first utilizes a characterbased bilstm model with unique features for each character, while the second uses a word-level bilstm layer and a subsequent dense layer with softmax activation. karim and abandah (2021) studied the effect of varying the training dataset size. each time, they trained a bilstm model and evaluated its performance. the results demonstrated that error rates improve as the size of the training corpus increases. al-rfooh et al. (2023) finetuned a token-free multilingual model called byt5 @xcite to perform arabic text diacritization as a sequenceto-sequence task, similar to the translation task. recently, skiredj and berrada (2024) introduced the pre-finetuned token classification for arabic diacritization (ptcad) model. this approach treats arabic text diacritization as a downstream task for a pretrained bert-like model. the approach starts with a pretraining phase on linguistically relevant tasks, such as part-of-speech (pos) tagging and segmentation, which are framed as masked language modelling (mlm) tasks. this pretraining helps enhance the model's contextual understanding. then, it moves into a finetuning phase where diacritization is handled as a token classification task. this phase leverages the contextual insights gained earlier to enhance diacritization accuracy. unlike skiredj and berrada's ( 2024 ) method, we consider a simpler approach where we directly pretrain a character-level bert model with no further modifications or extra labeling. ## experiments we pretrained a character-based bert model to find the effect of mlm pretraining on the diacritization performance. the model has 6 layers with d @xmath0. the model was trained using mlm loss for 6 epochs with a batch size of 512, using the data shown in for atd models, we selected two transformer architectures: encoder-decoder (ed) with 3 layers and encoder-only (eo) with 6 layers. we set the number of layers to 3 in the ed model to ensure comparability with the eo model in terms of the total number of layers. all our atd models were trained with the following configurations: figure 2: encoder-only (eo) model d @xmath1, number of @xmath2, batch @xmath3, and @xmath4. we used the adamw optimizer @xcite with a learning rate of 3 × 10 -5 and a weight decay of 1 × 10 -2. each model was trained on a single dedicated a100 gpu. all atd models in our experiments were trained for a maximum of 200 epochs with an early stopping criteria. generally, we define the input text x with a total number of characters t as a series of characters x 1 , x 2 , x 3 , . . . , x t where each character represents an undiacritized arabic letter. correspondingly, the output sequence y consists of diacritics y 1 , y 2 , y 3 , . . . , y t with each diacritic y i associated with the respective letter x i . in eo models, we express the relationship between letters and diacritics as follows: in other words, we predict the diacritic y i conditioned only on the input text x 1 , . . . , x t . in ed models, on the other hand, we consider the atd task as a translation task, where the input text represents the source language and the output diacritics sequence represents the target language. we express the relationship between letters and diacritics in ed models as follows: our experiments involved training a total of four models using both the ed and eo architectures. for both architectures, we used the pretrained character-based bert as the basis for initializing the weights. consequently, we obtained one ed model and one eo model with weights that reflect the knowledge encoded in the pretrained bert. additionally, we trained the other two models, consisting of one ed model and one eo model, where the weights were randomly initialized. these models allowed us to explore the impact of different weight initialization strategies on the performance and behavior of the models. for each model in our experiments, we selected two checkpoints. the first checkpoint was chosen after training the model for 5 epochs, while the second checkpoint was chosen as the best checkpoint achieved after training for a longer period. the best checkpoint of ed model was at epoch 175 while the best checkpoint of eo was at epoch 192. the purpose of this selection process was to study the impact of extended training duration on the models, even when they were exposed to the same amount of data. moreover, we randomly sampled 1m sentences from the pretraining dataset to be pseudo-labeled using the ns @xcite technique. we used the best checkpoints of both ed and eo models to pseudo-label two copies of the sampled data. finally, a new ed model as well as a new eo model were trained on tashkeela data combined with the pseudo-labeled 1m sentences. both models' parameters were initialized from the best checkpoints. ## results there are two methods to evaluate atd models: one with case ending (ce) and one without case ending (no ce). in the no ce approach, the diacritic on the last letter is excluded during performance evaluation, while the ce approach includes the diacritic in the evaluation. the presence or absence of the diacritic on the last letter mostly depends on grammatical rules. error rates without ce reflect the performance specifically on the core word, while error rates with ce represent the overall performance of the model (madhfar and qamar, 2020). we compared our models with 9 models, namely, cbhg (madhfar and qamar, 2020), sakhr foot_2 , farasa @xcite , d2 and d3 models (alkhamissi et al., 2020), alkhalil tashkeel foot_3 , mishkal foot_4 , multilevel diacritizer @xcite , and shakkala. moreover, we compared the performance of our models with the performance of two large language models (llms) on the atd task: gpt-4-turbo foot_5 , and command r+ foot_6 . during evaluation, all preprocessing steps are applied to both the reference text and the output of all models to ensure fair comparisons. in the case of long sentence diacritization, we follow a process of splitting the sentence into smaller segments based on punctuation. each small sentence is then diacritized individually, and finally, the segments are combined to reconstruct the original text with diacritics. the results in tables 4 and 5 show that our models achieved state-of-the-art performance in the atd task, outperforming all 11 models. our ed model achieved the lowest der and word error rate (wer) on the catt benchmark dataset with and without ce. moreover, both ed and eo models achieved low der and wer scores on the wikinews benchmark dataset compared to other atd models. the evaluation of gpt-4 on the wikinews dataset shows a high level of performance. however, when gpt-4 is evaluated on the catt dataset, its performance appears to be comparatively normal. this difference in performance can likely be attributed to the fact that gpt-4 was trained on web data predating december 2023. based on the observed significant performance gap between testing gpt-4 on catt and wikinews, we believe that gpt-4 was most likely trained on the wikinews dataset. it is worth noting that the wikinews dataset was published in 2017 @xcite , while the catt dataset was created in march 2024.