Improving A rabic Diacritization by Learning to Diacritize and Translate (2022)

Abstract: We propose a novel multitask learning method for diacritization which trains a model to both diacritize and translate. Our method addresses data sparsity by exploiting large, readily available bitext corpora. Furthermore, translation requires implicit linguistic and semantic knowledge, which is helpful for resolving ambiguities in diacritization. We apply our method to the Penn Arabic Treebank and report a new state-of-the-art word error rate of 4.79%. We also conduct manual and automatic analysis to better understand our method and highlight some of the remaining challenges in diacritization. Our method has applications in text-to-speech, speech-to-speech translation, and other NLP tasks.

## introduction arabic is typically written without short vowels and other pronunciation indication markers, 1 col lectively referred to as diacritics. a longstanding task in natural language processing (nlp) is to take undiacritized text and add the diacritics, re ferred to as diacritization (see figure 1 ). diacrit ics indicate both how to pronounce the word and resolve ambiguities in meaning between different words with the same (undiacritized) written form. diacritic prediction is the dominant source of errors in arabic grapheme to phoneme conver sion @xcite , a crucial component in many texttospeech and speechtospeech transla tion systems. diacritization also has applications in auto matic speech recognition (asr) @xcite , machine translation (mt) @xcite morphological analysis @xcite , lexical recognition tests @xcite zesch, ‫ﻟﻧذھب‬ ‫ھﯾﺎ‬ ‫بْ‬ َ ‫ھ‬ ْ ‫ذ‬ َ ‫ِﻧ‬ ‫ﻟ‬ ‫ﱠﺎ‬ ‫ﯾ‬ َ ‫ھ‬ [hjaː lnðhb] [hajːaː linaðhab] figure 1: arabic diacritization is the task of adding di acritics (markings above and below characters, shown in red) to arabic text. diacritics clarify how a word is pronounced, including short vowels and elongation, and disambiguate word meaning. here, we show the diacritization of ‫لنذهب‬ ‫هيا‬ (let's go). the ipa pronun ciations below each word demonstrate that the diacrit ics are crucial for pronouncing each word: the undia critized form maps to an incorrect pronunciation, while the diacritized form maps to the correct pronunciation (the contributions the diacritics make to the pronuncia tion are also shown in red). @xcite , and homograph resolution @xcite . we focus on modern standard arabic (msa), a standardized dialect of arabic used in most aca demic, legal, and news publications, and an ob vious choice for texttospeech (tts) systems. msa is the 5th most spoken foot_0 language in the world with about 274m speakers @xcite . ## method we train a single transformerbased @xcite encoderdecoder model to both trans late and diacritize, with the hypothesis that the translation task is complementary to diacritization. to maximize the number of shared representations between diacritized and undiacritized words, we train our model at the characterlevel. following work in multilingual mt, we prepend a tag to each output sentence to tell the model whether the out put is undiacritized arabic, diacritized arabic, en glish, french, or spanish during training. at infer ence time we force decode the tag to request that the model produce diacritized arabic. ## experiments we train a characterlevel transformer encoder decoder model on both diacritics data and the bitext. our primary model performs diacritiza tion, translation from arabic (ar) to english (en), french (fr), and spanish (es), and translation from english, french, and spanish to arabic. however, we also perform ablations for analysis purposes, leaving out (1) the @xmath0,fr,es} data, (2) the {en,fr,es}→ar data, and (3) all of the bitext data. each model uses a single encoder and decoder for all tasks. ## results the word error rates for our method (main model, both ablation models, and baseline) are shown in our ablation models also outperform all prior work, with the model trained on @xmath1,es,fr} (denoted @xmath2,es,fr}→ar (denoted * →ar) bi text, but neither perform as well as the main model trained on both @xmath3. (see §6 for more detailed comparisons between the models trained in this work.) finally, our baseline model, consisting of a characterbased transformer with no augmenta tion or word embeddings, slightly outperforms prior models from @xcite and @xcite , that also do not use mtl, morphological analyzers, or contextual em beddings. ## manual analysis to better understand the performance of our mtl model, we manually annotate all differences be tween our model prediction and the gold test set for a randomly selected 20% of the 1246 sentences in the test set that contain at least one disagreement. we find that approximately 66% of the disagree ments between the gold test set and the model are the result of model errors, which we denote as "true errors". the majority of these errors are due to case markings being either incorrect (38.6% of all true errors) or missing (16.5% of all true errors), while the rest of the word is correct. however, we find that in approximately 32% of disagreements the model output is, in fact, correct. we denote such cases as "false errors." about half (50.3%) of the false errors were due to the test set missing diacritics and another 31.2% of all false errors were due to errors in the test set diacritics. 10.7% of the false errors were the result of valid variations which did not change the meaning finally, about 2% of the disagreements are cases where the input to the model is not a real word, making the correct output undefined. ## conclusion we demonstrate that training a diacritics model to both diacritize and translate substantially outper forms a model trained on the diacritization task alone. adding translation data substantially in creases the amount of training data seen by the model, addressing data sparsity issues in diacriti zation. the translation task also injects semantic and linguistic knowledge into the model, helping the model resolve ambiguities in diacritization. our method achieves a new stateoftheart word error rate of 4.79% on the penn arabic tree bank datasets, using the standard data splits of @xcite . finally, we present extensive manual and au tomatic analysis which provides insight into our method and highlights several challenges that still remain in arabic diacritization, including proper nouns, female word forms, and case endings.