Embed More Ignore Less ( EMIL ): Exploiting Enriched Representations for A rabic NLP (2020)

Abstract: Our research focuses on the potential improvements of exploiting language specific characteristics in the form of embeddings by neural networks. More specifically, we investigate the capability of neural techniques and embeddings to represent language specific characteristics in two sequence labeling tasks: named entity recognition (NER) and part of speech (POS) tagging. In both tasks, our preprocessing is designed to use enriched Arabic representation by adding diacritics to undiacritized text. In POS tagging, we test the ability of a neural model to capture syntactic characteristics encoded within these diacritics by incorporating an embedding layer for diacritics alongside embedding layers for words and characters. In NER, our architecture incorporates diacritic and POS embeddings alongside word and character embeddings. Our experiments are conducted on 7 datasets (4 NER and 3 POS). We show that embedding the information that is encoded in automatically acquired Arabic diacritics improves the performance across all datasets on both tasks. Embedding the information in automatically assigned POS tags further improves performance on the NER task.

## introduction named entity recognition (ner) and part-of-speech (pos) tagging have traditionally been used as preprocessing steps in many natural language processing (nlp) applications. for example, @xcite discussed the use of ner across question answering, information retrieval, co-reference resolution, topic modeling, and machine translation. similarly, pos tagging is often applied early in the nlp pipeline for many applications including information retrieval systems, syntax, and semantic analysis, speech recognition systems and machine translation @xcite . in recent years, arabic has been studied increasingly due to the explosion in the number of arabic users on social media and the internet in general. arabic is a morphologically rich language with complex grammatical structure @xcite . arabic nlp researchers have used two types of approaches and sometimes a mixture of both to work with arabic text. the first approach is the simplification approach where researchers tend to apply preprocessing (transformation) that simplify arabic text such as letter normalization @xcite and transliteration @xcite . the second approach is the enrichment approach where researchers tend to apply minimum modification to the arabic text and devise a way of incorporating the enriched features and potentially add more features to it. we assume that the simplification approach may exclude some useful information, and hence take an enrichment approach. first, we add diacritics information inferred by automatic diacritization model called shakkala, based on the assumption that the syntactic and semantic information encoded in diacritics might be useful in both the ner and the pos task. once we have a diacritic-enhanced pos model, we use it to infer pos information for the ner corpora, based on the assumption that both diacritics and pos information can potentially improve the performance of the ner model. we are aware that this pipeline will raise the question of the quality of the inferred information and its effect on performance. nevertheless, the experimental results shows that the addition of this automatically-inferred information enhances the performance. our specific contributions are that we propose a framework (emil) where we embed more and ignore less. we show that applying minimal modification to the text and embedding more of the possible features into the model can outperform the standard sequence labeling models. we propose a diacritic-aware architecture for sequence labeling which extends and outperforms the current standard character-aware architecture of a bi-directional long short-term memory network (bi-lstm) with a crf. we also propose a combination architecture for ner that combines word, character, diacritic, and pos information and outperforms the standard character-aware architecture and our own diacritic-aware architecture. we will now discuss the main characteristics of the arabic language section (2.1) and related work on arabic ner and pos tagging section (2.2). ## hypothesis the hypothesis of this research can be summarized thus: we have already outlined in section (1) that there are two approaches of handling arabic text, the simplification approach which involves transforming the text into simplified representation and the enrichment approach which minimally modifies the text and potentially adds more exploitable features to the text. we have assumed that the simplification approach might exclude some useful information and hence have adopted an enrichment approach. more specifically, we have adopted a pipeline approach where we use one model to infer a particular linguistic characteristic and then use the inferred information as features further downstream. for example, we infer diacritic information, using existing models such as shakkala. we then exploit these features of the text in the form of embeddings in order to enhance performance in pos-tagging. we then use diacritics inferred by shakkala and pos information inferred by pos model to enhance performance in ner. we reason that using minimally modified arabic text as input to neural networks builds the potential for allowing the neural network to learn an enriched representation of the arabic language. further, a framework such as emil, which incorporates more language-specific characteristics of the text in the form of embeddings should result in improved performance. in particular, since arabic syntax and word sense disambiguation relies heavily on diacritics, we reason that applying an automatic diacritization neural model to minimally transformed arabic text can capture syntactic and semantic dependencies. similarly, useful information for ner can also be derived via pos tagging. finally, we hypothesize that incorporating embedding layers based on derived language characteristics such as diacritics and pos tags can improve the overall performance of a neural network in sequence labeling tasks. ## approach we propose a three-step approach to arabic sequence labeling. the first step is to automatically diacritize the text using the state-of-the-art automatic diacritization system shakkala @xcite . the second step is the individual training of character and diacritic embeddings using the architecture proposed by @xcite . the third step is to train all embedding layers together using a combination model (see section 4.3). there are two main advantages in adopting this architecture for emil. first, it is based on a standard approach in ner and sequence labeling in general, which remains very close to the state-ofthe-art. second, it is a relatively light-weight architecture requiring less computational resources than other alternatives (see section 5.3). we will discuss and justify our design choices and the computational aspects of the architecture further in section (5.3) and section (6). figure (2 ) gives an overview of the overall training procedure for our emil framework, which we now explain in detail. ## empirical evaluation the character model described previously is used both as a baseline and for training the individuallytrained character embedding. we also empirically evaluate two versions of the combination model: the character-diacritic model which incorporates word, character, and diacritic embedding layers; and the four-layer combination model which has an additional pos layer. we evaluate our models on two sequence labeling tasks: ner and pos tagging. we now describe the datasets used section (5.1), hyperparameter settings section (5.2) and the experiments performed to test our hypotheses section (5.3). we used k-fold cross validation in order to evaluate the statistical significance of our results. the value of k used in k-fold cross validation varied according to the size of the dataset but was between 4 and 10 in all cases. ## results first, we compare results for the character model and the character-diacritic model to see the impact of adding diacritics. we observe that the proposed character-diacritic model outperforms the character model on all of the datasets for both ner and pos-tagging. automatically diacritizing the text and adding the diacritic embedding layer boosts both precision and recall of the system. we used k-fold cross-validation and all of the differences observed are statistically significant at the 5% level using a paired sample test. our best absolute performance in ner was on the binajeeba dataset where we achieved an f1-score of 0.84, substantially outperforming the character model which achieved 0.72. our best absolute and relative gain in performance on pos-tagging was on the al mushaf dataset where f1 performance rose from 0.941 to 0.980. second, we compare the character-diacritic model with the four-layer combination model to evaluate the impact of adding pos information as well as diacritics. we note that automatically pos tagging the diacritized ner datasets and adding this information via a pos embedding layer leads to a substantial gain in performance on both datasets. the performance on the binajeeba dataset improved form 0.84 to 0.93 using the four-layer combination model. similarly, f1-score performance on the anercorp dataset rose from 0.82 to 0.92. we also performed a number of ablation experiments on one of the benchmark datasets (anercorp) which are summarised in table ( 2 ). it is worth noting that all the embedding layers in the ablation experiments are trainable apart from a5. in our ablations, we consider the effect of randomly initialising or training the different embedding layers for the combination model as follows: (a1) all embeddings randomly initialised, this ablation shows the importance of the pre-trained and the individual-trained components of the combination model. as shown in the table the performance drops substantially when we remove these components; (a2) word embeddings and pos randomly initialised, individually-trained character and diacritic embeddings, this ablation shows the significance of the pre-trained word embeddings and evidently proves its impact on the model. when we use randomly initialized word embeddings the performance drop from 0.929 to 0.611; (a3) pre-trained word embeddings, randomly initialized character, diacritics and pos embeddings, this ablation shows the impact of removing the individual training step from the combination model. the result shows that without the individual training of character and diacritic embedding the model achieves 0.747 however, when initializing character and diacritic embeddings of the combination model with individually-trained weights which are trained separately the performance is boosted to 0.929; and (a4) pre-trained word embeddings and the rest are individually trained, this ablation is to test the effectiveness of individually training pos embeddings and the results shows that this approach still outperformed by the original approach where we use randomly initialized pos embedding. we also considered (a5) freezing the embeddings rather than allowing them to be trainable in the final model but our results show the massive benefit of trainable embeddings. the previous ablation experiments supports the choice of our architecture where we use pre-trained word embedding along with randomly initialized pos embedding and individually trained character and diacritic embeddings. ## error analysis and discussion we now present a detailed analysis of the errors committed by each model on the anercorp dataset for the ner task. in our analysis, we identified 4 types of errors. type-one: boundary errors e.g., the word (detroit) appears once in the dataset tagged as b-org and is classified as i-org. typetwo: low frequency words e.g., the word (isaac) appears twice in the dataset tagged as b-pers and is classified as o. we note that a lot of words in this type of error are foreign names. type-three: dominant tags e.g., the word (the ambassador) is tagged twice as b-org and 12 times as o, and when this word appears as b-org a model classifies it as o. type-four: counter dominant tags e.g., the word (mashhad) is labeled 17 times as b-loc and once as o, and when it appears as b-loc a model misclassifies it as o. we note that a lot of type-four errors appear to be gold standard errors where the word is tagged incorrectly and classified correctly by the model. for example, the word (al daffa (the bank, like in riverbank)) is incorrectly tagged 4 times as o in the gold standard and our model classified it as b-loc, which is in fact the correct label. table (3) shows a summary of the errors. we observe that the addition of diacritics reduced the number of errors in each type. further, each error type was reduced again by the addition of pos information. however, the largest reductions were for type-three and type-four errors which are both due to tag ambiguity. for example, the word (al saudia) can be tagged either as i-loc or b-loc depending on the context. adding diacritics to this word will help disambiguate its sense. for example instances which should be tagged as b-loc are adverbs and will be diacritized as (al saudiate) ## conclusions and future work in summary, we have shown that inferring and incorporating linguistic features of arabic text in the form of embeddings can improve the performance of downstream tasks. further, the pipeline approach that we are using may not be ideal regarding the quality of the inferred information but it still produces excellent results overall. focussing on diacritic information, we have shown that it is possible for a neural network to learn from the information encoded in the diacritics of the arabic text and for this information to be used successfully in pos tagging and ner. by automatically diacriticising text and adding a diacritic-aware layer to existing neural architecture, performance (f1) was increased. we have also shown that adding pos information on top of diacritics in a similar way further improves performance at ner. there are a number of directions for further work. first, further work on automatic diacritization, potentially directly including the diacritic inference model into the ner/pos training mode to reduce the effect of error propagation cause by the pipeline. second, there is work to be done investigating the impact of embedding diacritic and pos information in attention-based architectures @xcite @xcite . second, we believe that our approach of diacritic sensitive tagging will be useful in other areas of analysis including segmentation and deeper syntactic analysis. further, embedding information from other analyses, including grammatical dependencies, might further improve performance in downstream tasks such as ner. consequently, this is clearly only the beginning of investigating how the emil approach, where we embed more and ignore less, can be applied to different sources of information, in different languages and to different downstream tasks.