G ulf A rabic Diacritization: Guidelines, Initial Dataset, and Results (2022)

Abstract: Arabic diacritic recovery is important for a variety of downstream tasks such as text-to-speech. In this paper, we introduce a new Gulf Arabic diacritization dataset composed of 19,850 words based on a subset of the Gumar corpus. We provide comprehensive set of guidelines for diacritization to enable the diacritization of more data. We also report on diacritization results based on the new corpus using a Hidden Markov Model and character-based sequence to sequence models.

## introduction arabic has two types of vowels, namely long and short vowels. although long vowels are explicitly written, short vowels, which take the form of diacritic marks, are typically omitted in written arabic, and readers need to infer these diacritics to properly pronounce words. thus, diacritic recovery, also referred to as diacritization, is important for downstream tasks such as text-to-speech and language learning. most previous efforts pertaining to arabic diacritic recovery have focused on modern standard arabic (msa) and classical arabic (ca). the focus on these arabic varieties has been aided by the availability of large training corpora such as the penn arabic treebank @xcite and tashkeela @xcite and relatively stable diacritization standards. there has been some efforts related to diacritizing different arabic dialects such as egyptian @xcite , palestinian @xcite , morrocan, and tunisian @xcite . many challenges face dialectal diacritization, mostly related to the availability of large consistent data. while msa/ca corpora may be composed of millions of words, dialectal datasets have been capped at tens of thousands of words @xcite . furthermore, diacritization of the same dialect may differ from town to town, complicating data standardization and consistency. for example, the word (sxn foot_0 -hot) is diacritized in the egyptian dialect as (suxuno) in alexandria and (suxono) in cairo. variations in pronunciation of words are rather common within the same dialect in locales of close geographical proximity. in this paper, we present a new public diacritized dataset for gulf arabic in accordance to the pronunciation of the city of dubai in the united arab emirates (uae). the dataset is a 19,850 words subset of the gumar corpus @xcite , which is composed of roughly 200 thousand words from emirati internet novels. to diacritize the cor-pus, we conducted a workshop that included two senior computational linguists and 15 native speakers of the emirati dialect to codify the diacritization guidelines and to actually diacritize the corpus. we split the corpus into training and test sentences, and we proceeded to build two different emirati diacritizers using a word-based hidden markov model (hmm) and a character-based sequence to sequence mapping architecture. the contributions of this paper is as follows: many approaches have been used for arabic diacritization such as hmms @xcite , finite state transducers @xcite , character-based maximum entropy based classification @xcite , and a variety of deep learning approaches @xcite @xcite . for msa, most approaches tend to handle core word diacritics, which disambiguate a word in context, separately from case-endings, which typically appear at end of a word and specify the syntactic role of words. however, more recent approaches have resorted to guessing both types of diacritics jointly @xcite by either casting the problem as a character sequence labeling problem or as a character sequence to sequence (seq2seq) mapping respectively. since the seq2seq models have a tendency to hallucinate, @xcite used a combination of limited context and voting to overcome the problem. unlike msa, arabic dialects generally omit case-endings and tend to apply sukun (o) on the last letter. prior work on dialectal diacritization is rather scant. for dialectal egyptian, zalmout and habash (2020) developed a morphological analyzer that also performs diacritization using sequence to sequence modeling. they reported a diacritization accuracy of 85.0%. for dialectal @xcite developed a morphological analyzer for dialectal gulf verbs, but diacritiza-tion was not the focus of their work. @xcite morphologically tagged an emirati subset of the gumar corpus including the diacritization of lemmas. however, mapping the diacritics from lemmas to words is non-trivial. for dialectal palestinian, @xcite annotated a corpus of containing 43k words and diacritized all words. abdelali et al. (2018); darwish et al. (2018); mubarak et al. (2019) used diacritized translations of the bible into dialectal moroccan (151k words) and tunisian (142k words) to train bilstm over crf, crf only, and seq2seq models respectively for diacritizing both dialects. of all three approaches, the seq2seq model led to the lowest word error rate (moroccan: 1.4% and tunisian: 2.5%). ## dataset as mentioned earlier, we diacritized a subset of the gumar corpus @xcite . the gumar corpus is a collection of internet novels composed of roughly 100 million words. a 200 thousand words subset of gumar was in the emirati dialect and was manually morphologically tagged. though the lemmas were diacritized, their diacritization often did not correspond directly to the diacritization of words. thus, we proceeded to diacritize a 19,850 word subset of the tagged emirati portion of gumar. we used the codafied version of the text, as opposed to the raw text, to have greater consistency in spelling. coda, or conventional orthography for dialectal arabic, is an attempt at standardizing the spelling of different arabic dialects @xcite . for diacritization, we conducted a workshop that included two senior computational linguists and 15 native speakers of the emirati dialect to codify the diacritization guidelines. diacritization standards: after lengthy discussions, we settled on the following guidelines: diacritization process: we used a three step diacritization process designed to increase speed and improve accuracy. the steps are as follows: ## experiments we trained two different diacritization models based on our new dataset. prior to training the models, we tokenized all the text to separate all punctuation. the data did not have any emojis, urls, or emails. the models were as follows: hmm model: as the name suggests, we used a hidden markov model to find the best diacritization of words in context. we used kenlm foot_2 to train a word trigram language model and an in-house implementation of a-star search to ascertain the best path in the lattice. ## conclusion in this paper we introduced a new dataset for gulf arabic diacritization based on the sub-dialect spoken in dubai, uae. the diacritization was based on formalized diacrtization guidelines that was developed by two senior computational linguists along with 15 native speakers, who were also instrumental in performing the actual diacritization. we plan to release the dataset publicly under an open source license. we also presented initial results using 2 different diacritization models. though the dataset is relatively small (19,850 words), we were able to build two diacritization models that achieved less than 9% word error rate. we plan to expand the size of the corpus, particularly for non-codafied gulf text. we hope that models trained on our data can help significantly speed up the diactization process.