Deep Diacritization: Efficient Hierarchical Recurrence for Improved A rabic Diacritization (2020)

Abstract: We propose a novel architecture for labelling character sequences that achieves state-of-the-art results on the Tashkeela Arabic diacritization benchmark. The core is a two-level recurrence hierarchy that operates on the word and character levels separately—enabling faster training and inference than comparable traditional models. A cross-level attention module further connects the two and opens the door for network interpretability. The task module is a softmax classifier that enumerates valid combinations of diacritics. This architecture can be extended with a recurrent decoder that optionally accepts priors from partially diacritized text, which improves results. We employ extra tricks such as sentence dropout and majority voting to further boost the final result. Our best model achieves a WER of 5.34%, outperforming the previous state-of-the-art with a 30.56% relative error reduction.

## introduction the arabic script (and similarly hebrew, aramaic, pahlavi...) is an impure abjad. these writing systems represent short consonants and long vowels using full letter graphemes, but generally omit short vowels and consonant length from writing. this leaves the task of inferring the missing phonemes to the reader by using context from neighbouring words and knowledge of the language structure to determine the correct pronunciation and disambiguate the meaning of the text. those sounds are represented by diacritical marks-small graphemes that appear usually above or below a basic letter in the abjad. we focus in this work on diacritization of arabic texts. however, our proposed architecture has no explicitly language-dependent components and should be adaptable for other character sequence labelling tasks. although it is the first language of several million people, and is spoken in some of the fastest growing markets @xcite , the arabic language, like many others, lacks attention from the nlp community compared to established test bed languages such as english or chinese, which both enjoy higher momentum and an abundance of established resources and techniques. the automatic restoration of diacritics to arabic text is arguably one of the most important nlp tasks for the arabic language. besides direct applications like facilitating learning, diacritics are used to enhance language modeling, acoustic modeling for speech recognition, morphological analysis, machine translation, and text-to-speech systems (which need to restore the lost phonemes to render words properly) @xcite . to illustrate this further, ## related work the literature surrounding the automatic diacritization of arabic text provides methods in two categories: classical rule-based solutions, and statistical modeling-based methods. early approaches have worked on constructing a large set of language specific rules to restore the lost diacritics @xcite @xcite . researchers have then shifted to rely more on learning-based methods that do not require extra expert systems such as morphological analyzers and part-of-speech taggers. @xcite have shown that recurrent neural networks are suitable candidate models for learning the task entirely from data and can be easily extended to other languages and dialects without the use of manually engineered features. other methods such as hidden markov models (hmms) @xcite , conditional random fields (crfs) @xcite , maximum-entropy models @xcite and finite-state transducers @xcite have similarly been employed. however, more recent works have started to use deep (neural-based) architectures such as sequence-to-sequence transformers and recurrent cell-based models inspired by work in neural machine translation @xcite . solutions combining both rule-based and deep learning methods appear in recently published work @xcite . @xcite have shown that the diacritization task benefits from jointly modelling lexicalized and non-lexicalized morphological features instead of targeting only the diacritization task. ## conclusion in this work, we presented a novel architecture that outperforms previously published results on the tashkeela arabic diacritization benchmark. future work may include: