Improving A rabic Diacritization with Regularized Decoding and Adversarial Training (2021)

Abstract: Arabic diacritization is a fundamental task for Arabic language processing. Previous studies have demonstrated that automatically generated knowledge can be helpful to this task. However, these studies regard the auto-generated knowledge instances as gold references, which limits their effectiveness since such knowledge is not always accurate and inferior instances can lead to incorrect predictions. In this paper, we propose to use regularized decoding and adversarial training to appropriately learn from such noisy knowledge for diacritization. Experimental results on two benchmark datasets show that, even with quite flawed auto-generated knowledge, our model can still learn adequate diacritics and outperform all previous studies, on both datasets.

## introduction modern standard arabic (msa) is generally written without diacritics, which poses a challenge to text processing and understanding in downstream applications, such as text-to-speech generation @xcite and reading comprehension @xcite . restoration of such diacritics, known as diacritization, becomes an important task for arabic natural language processing (nlp). among different diacritization methods @xcite @xcite @xcite , the neural ones @xcite @xcite @xcite achieve the best performance due to their better capability in incorporating contextual features. to further improve diacritization, automatically generated knowledge from off-the-shelf toolkits, such as morphological features, parts-of-speech tags, and automatic diacritization results, have been extensively applied to this task @xcite @xcite . however, current models treat such knowledge instances as gold references and always directly concatenate them with input embeddings @xcite , which may lead to inferior results since the knowledge may be inaccurate, especially if the toolkits were trained on data with different criteria. diacritization can be performed by characterbased sequence labeling @xcite @xcite . we follow this paradigm and propose a neural approach in this paper, using regularized decoding and adversarial training to incorporate auto-generated knowledge (i.e., the diacritization results generated from off-the-shelf toolkits). specifically, the regularized decoder treats the auto-generated knowledge as separate gold labels and learns to predict them in a separate decoding process, which is then used to update the main model. the adversarial training is applied to the encoding process by determining whether the diacritization for an input follows the gold label or the auto-generated knowledge. in doing so, our model can dynamically distinguish between auto-generated knowledge instances instead of treating them all as gold references, so as to effectively identify what knowledge should be leveraged for different inputs. importantly, regularized decoding and adversarial training are exclusively applied to the training stage; we only need the main tagger for inference once the model has been trained. experimental results and further analyses illustrate the effectiveness of our approach, where our model outperforms strong baselines and achieves state-of-the-art results on two benchmark datasets: arabic treebank (atb) @xcite and tashkeela @xcite . ## the proposed approach as shown in figure 1 , our approach for diacritization follows the sequence labeling paradigm, where it has two training stages for the main tagger (m). in the first training stage (presented in the orange box in figure 1 ), m is enhanced by regularized decoding (rd) and adversarial training (at ) to discriminatively learn from the auto-generated labels. specifically, given an input arabic character sequence @xmath0, m and rd aim to predict two types of diacritization labels, y and y k , which follow the gold and auto-generated label criteria, respectively. at ensures that the main tagger only learns useful information from either gold or auto-generated labels. therefore, the first training stage can be conceptually formalized by afterwards, in the second training stage (presented in the green box in figure 1 ), m is further trained alone on the gold labels y without using auto-generated y k , rd and at , to fine-tune its parameters, where all parameters in se obtained through the first training stage are fixed. for inference, only m is used without requiring any additional input other than x to obtain the diacritization results. in the following sections, we first describe m, then elaborate the details of rd and at . ## conclusion in this paper, we propose to incorporate autogenerated knowledge (diacritization labels in another criterion) for arabic diacritization with regularized decoding and adversarial training. in detail, the regularized decoding treats the auto-generated knowledge as separate "gold" labels and learns to predict them in another decoding process; the adversarial training is used to ensure that the shared information from gold and auto-generated labels are learned to help diacritization. with the regularized decoding and adversarial training, the main tagger in our approach is able to smartly leverage auto-generated knowledge provided by an existing diacritization tagger. experimental results on two benchmark datasets illustrate the validity and effectiveness of our model, where state-of-the-art performance is obtained on both datasets.