A Multitask Learning Approach for Diacritic Restoration (2020)

Abstract: In many languages like Arabic, diacritics are used to specify pronunciations as well as meanings. Such diacritics are often omitted in written text, increasing the number of possible pronunciations and meanings for a word. This results in a more ambiguous text making computational processing on such text more difficult. Diacritic restoration is the task of restoring missing diacritics in the written text. Most state-of-the-art diacritic restoration models are built on character level information which helps generalize the model to unseen data, but presumably lose useful information at the word level. Thus, to compensate for this loss, we investigate the use of multi-task learning to jointly optimize diacritic restoration with related NLP problems namely word segmentation, part-of-speech tagging, and syntactic diacritization. We use Arabic as a case study since it has sufficient data resources for tasks that we consider in our joint modeling. Our joint models significantly outperform the baselines and are comparable to the state-of-the-art models that are more complex relying on morphological analyzers and/or a lot more data (e.g. dialectal data).

## introduction in contrast to english, some vowels in languages such as arabic and hebrew are not part of the alphabet and diacritics are used for vowel specification. 1 in addition to pertaining vowels, diacritics can also represent other features such as case marking and phonological gemination in arabic. not including diacritics in the written text in such languages increases the number of possible meanings as well as pronunciations. humans rely on the surrounding context and their previous knowledge to infer the meanings and/or pronunciations of words. however, computational models, on the other hand, are inherently limited to deal with missing diacritics which pose a challenge for such models due to increased ambiguity. diacritic restoration (or diacritization) is the process of restoring these missing diacritics for every character in the written texts. it can specify pronunciation and can be viewed as a relaxed variant of word sense disambiguation. for example, the arabic word elm foot_2 can mean "flag" or "knowledge", but the meaning as well as pronunciation is specified when the word is diacritized ( ealamu means "flag" while eilomo means "knowledge"). as an illustrative example in english, if we omit the vowels in the word pn, the word can be read as pan, pin, pun, and pen, each of these variants have different pronunciations and meanings if it composes a valid word in the language. the state-of-the-art diacritic restoration models reached a decent performance over the years using recurrent or convolutional neural networks in terms of accuracy @xcite @xcite and/or efficiency @xcite ; yet, there is still room for further improvements. most of these models are built on character level information which help generalize the model to unseen data, but presumably lose some useful information at the word level. since word level resources are insufficient to be relied upon for training diacritic restoration models, we integrate additional linguistic information that considers word morphology as well as word relationships within a sentence to partially compensate for this loss. in this paper, we improve the performance of diacritic restoration by building a multitask learning model (i.e. joint modeling). multitask learning refers to models that learn more than one task at the same time, and has recently been shown to provide good solutions for a number of nlp tasks @xcite . the use of a multitask learning approach provides an end-to-end solution, in contrast to generating the linguistic features for diacritic restoration as a preprocessing step. in addition, it alleviates the reliance on other computational and/or data resources to generate these features. furthermore, the proposed model is flexible such that a task can be added or removed depending on the data availability. this makes the model adaptable to other languages and dialects. we consider the following auxiliary tasks to boost the performance of diacritic restoration: word segmentation, part-of-speech (pos) tagging, and syntactic diacritization. we use arabic as a case study for our approach since it has sufficient data resources for tasks that we consider in our joint modeling. 3 the contributions of this paper are twofold: ## diacritization and auxiliary tasks we formulate the problem of (full) diacritic restoration (diac) as follows: given a sequence of characters, we identify the diacritic corresponding to each character in that sequence from the following set of diacritics {a, u, i, o, k, f, n, ∼, ∼a, ∼u, ∼i, ∼f, ∼k, and ∼n}. we additionally consider three auxiliary tasks: syntactic diacritization, partof-speech tagging, and word segmentation. two of which operate at the word level (syntactic diacritization and pos tagging) and the remaining tasks (diacritic restoration and word segmentation) operate at the character level. this helps diacritic restoration utilize information from both charac-ter and word level information, bridging the gap between the two levels. ## approach we built a diacritic restoration joint model and studied the extent to which sharing information is plausible to improve diacritic restoration performance. our joint model is motivated by the recent success of the hierarchical modeling proposed in @xcite ) such that information learned from an auxiliary task is passed as input to the diacritic restoration related layers. foot_7 ## results and analysis we use wordtochar representation rather than characters for all remaining models that jointly learn more than one task. for all experiments, we observe improvements compared to both baselines across all evaluation metrics. furthermore, all models except diac+seg outperform wordtochar diacritic restoration model in terms of wer, showing the benefits of considering output distributions for the other tasks. despite leveraging tasks focused on syntax (syn/pos) or morpheme boundaries (seg), the improvements extend to lexical diacritics as well. thus, the proposed joint diacritic restoration model is also helpful in settings beyond word final syntactic related diacritics. the best performance is achieved when we consider all auxiliary tasks within the diacritic restoration model. impact of auxiliary tasks: we discuss the impact of adding each investigated task towards the performance of the diacritic restoration model. word segmentation (diac+seg): when morpheme boundaries as well as diacritics are learned jointly, the wer performance is slightly reduced on all and oov words. this reduction is attributed mostly to lexical diacritics. as arabic exhibits a non-concatenative fusional morphology, reducing its complexity to a segmentation task might inherently obscure morphological processes for each form. observing only slight improvement is surprising; we believe that this is due to our experimental setup and does not negate the importance of having morphemes that assign the appropriate diacritics. we speculate that the reason for this is that we do not capture the interaction between morphemes as an entity, losing some level of morphological information. for instances, the words waham∼a versus wahum for the undiacritized words whm (bold letters refer to consonants distinguishing it from diacritics) would benefit from morpheme boundary identifications to tease apart wa from hum in the second variant (wahum), emphasizing that these are two words. but on the other hand, it adds an additional layer of ambiguity for other cases like the morpheme ktb in the diacritic variants kataba, kutubu, sayakotubo -note that the underlined segment has the same consonants as the other variantsin which identifying morphemes increased the number of possible diacritic variants without learning the interactions between adjacent morphemes. furthermore, we found inconsistencies in the dataset for morphemes which might cause the drop in performance when we only consider seg. when we consider all tasks together, these inconsistencies are reduced because of the combined information from different linguistic signals towards improving the performance of the diacritic restoration model. ## related work the problem of diacritization has been addressed using classical machine learning approaches (e.g. maximum entropy and support vector machine) @xcite or neural based approaches for different languages that include diacritics such as arabic, vietnamese, and yoruba. neural based approaches yield stateof-the-art performance for diacritic restoration by using bidirectional lstm or temporal convolutional networks @xcite @xcite . arabic syntactic diacritization has been consistently reported to be difficult, degrading the performance of full diacritic restoration @xcite @xcite @xcite . to improve the performance of syntactic diacritization or full diacritic restoration in general, previous studies followed different approaches. some studies separate lexical from syntactic diacritization @xcite . other studies consider additional linguistic features such as pos tags and word segmentation (i.e. tokens or morphemes) @xcite @xcite . @xcite addresses syntactic diacritization by building bilstm model in which its input embeddings are augmented with manually generated features of context, pos tags, and word segments. @xcite use deep belief network to build a diacritization model for arabic that focuses on improving syntactic diacritization and build subclassifiers based on the analysis of a confusion matrix and pos tags. regarding incorporating linguistic features into the model, previous studies have either used morphological features as a preprocessing step or as a ranking step for building diacritic restoration models. as a preprocessing step, the words are converted to their constituents (e.g. morphemes, lemmas, or n-grams) and then diacritic restoration models are built on top of that @xcite . @xcite use pos tags to improve diacritic restoration at the syntax level assuming that pos tags are known at inference time. as a ranking procedure, all possible analyses of words are generated and then the most probable analysis is chosen @xcite @xcite . @xcite develop a morphological disambiguation model to determine arabic morphological features including diacritization. they train the model using bilstm and consult with a lstm-based language model as well as other morphological features to rank and score the output analysis. similar methodology can be found in @xcite but using support vector machines. this methodology shows better performance on out of vocabulary (oov) words compared to pure character models. ## discussion & conclusion we present a diacritic restoration joint model that considers the output distributions for different related tasks to improve the performance of diacritic restoration. our results shows statistically significant improvements across all evaluation metrics. this shows the importance of considering additional linguistic information at morphological and/or sentence levels. including semantic information through pretrained word embeddings within the diacritic restoration model also helped boosting the diacritic restoration performance. although we apply our joint model on arabic, this model provides a framework for other languages that include diacritics whenever resources become available. although we observed improvements in terms of generalizing beyond observed data when using the proposed linguistic features, the oov performance is still an issue for diacritic restoration.