A ra DIC : A rabic Document Classification Using Image-Based Character Embeddings and Class-Balanced Loss (2020)

Abstract: Classical and some deep learning techniques for Arabic text classification often depend on complex morphological analysis, word segmentation, and hand-crafted feature engineering. These could be eliminated by using character-level features. We propose a novel end-to-end Arabic document classification framework, Arabic document image-based classifier (AraDIC), inspired by the work on image-based character embeddings. AraDIC consists of an image-based character encoder and a classifier. They are trained in an end-to-end fashion using the class balanced loss to deal with the long-tailed data distribution problem. To evaluate the effectiveness of AraDIC, we created and published two datasets, the Arabic Wikipedia title (AWT) dataset and the Arabic poetry (AraP) dataset. To the best of our knowledge, this is the first image-based character embedding framework addressing the problem of Arabic text classification. We also present the first deep learning-based text classifier widely evaluated on modern standard Arabic, colloquial Arabic, and Classical Arabic. AraDIC shows performance improvement over classical and deep learning baselines by 12.29% and 23.05% for the micro and macro F-score, respectively.

## introduction arabic is one of the six official languages of the united nations and the official language of 26 states. it is spoken by as many as 420 million people making it the fifth most popular language worldwide. according to the internet world statistics, as of 2017, arab users represent 4.8% of internet users 1 . arabic can be classified into three different types each having its own purpose and morphology. the modern standard arabic, the colloquial or dialectal 1 arabic speaking internet users and population statistics. https://www.internet-worldstats.com/ stats19.html accessed: 16-dec-2018, arabic and the classical or old arabic. the modern standard arabic is the official language used in media, government, news papers and is taught in schools. colloquial arabic varies between countries and regions. old or classical arabic survives nowadays in religious scriptures and old poetry. arabic has 28 basic letters all are consonants except three, which are long vowels. arabic is written from right to left. most arabic letters have more than one written form depending on their position in the word. for example, " ". " ", " ", and " " are all different forms of the letter " "(sīn). in addition, diacritical marks/short vowels that contribute to the phonology of arabic, greatly alter the character shape. example, " ", " ", " ", " ", " ", " ", " ", and " " are combination of the letter " "(bā') with different diacritics. this visual nature of the arabic letters is the main motivation for us to use image based embeddings. the importance of text classification has increased due to the increase of textual data on the internet as a result of social networks and news sites. common examples of text classification are sentiment analysis @xcite , spam detection (el-halees, 2009) and news categorization @xcite . arabic text classification is particularly challenging because of its complex morphological analysis. most research on arabic text classification has used classical techniques for feature extraction @xcite , which require complex morphological analysis, such as negation handling @xcite , part of speech tagging @xcite , stemming @xcite , and segmentation @xcite . arabic segmentation is especially complex because arabic words are not always separated by white spaces. it also includes some hand-crafted features like document term matrix with term frequency inverse document frequency (tf-idf) scores or word count. arabic text classification have been often done using classical algorithms like support vector machines (svms) or naive bayes @xcite . despite advances of text classification using deep learning techniques, little work has been done on arabic. @xcite introduced aravec, which is a pretrained distributed word embeddings @xcite . they trained their model using the skip-gram and continuous bag of words techniques. they used data from different sources like wikipedia and twitter. more recently, sagheer and sukkar (2018) used aravec's pretrained word embeddings with sentence convolutional neural network (cnn) originally proposed by @xcite for arabic document classification. this method still did not mitigate the problem of arabic word segmentation. those combinations left two major issues unaddressed. first, performance highly depends on morphological analysis and word segmentation, which is difficult for arabic. the same problem has been addressed for languages such as japanese and chinese @xcite . second, obtaining appropriate embedding (i.e. building hand-crafted features) is difficult. to solve these problems, character-based approaches utilizing deep learning methods mainly used in image processing have been proposed @xcite @xcite . @xcite introduced a character-level cnn (clcnn) that treats text as a raw signal at character level. the cnn then learns the language morphology and extracts appropriate features for text classification. their method mitigated the issue of complex morphological analysis. after that, @xcite proposed imagebased character embeddings for japanese and chinese text classification. their model was composed of a convolutional auto-encoder (cae) @xcite and a clcnn. they were the first to handle a character as an image and obtained character-embedding with their cae. they also introduced wild card training as a data augmentation technique, which is dropout @xcite on the embedding space. later, @xcite used image-based character embeddings learned through a character en-coder (ce) to train a gated recurrent unit (gru) for japanese, chinese, and korean text classification. @xcite proposed ce-clcnn that concatenated @xcite 's ce with clcnn as an end-to-end system and introduced random erasing on image domain as a data augmentation method. these models using character-level features learn language morphology eliminating the need for complex morphological analysis and word segmentation. another problem is that large text classification datasets usually suffer from long tailed data distribution problem. this means that few classes make up majority of data. this problem often reduces the model's accuracy on the minority classes making more biased towards majority classes. this problem can be addressed by either resampling @xcite @xcite @xcite or re-weighting the cost function @xcite @xcite @xcite . @xcite noticed that re-weighting the cost function by inverse class frequency as used in vanilla schemes @xcite @xcite could lead to poor performance on majority classes. they proposed class-balanced (cb) loss based on the effective number of classes which re-weights the loss by the inverse of the effective number of classes. our contributions can be summarized as follows: to the best of our knowledge, this is the first time an image-based character embedding model is used for arabic text classification. also, the first time a deep-learning based model is tested on datasets containing the three types of arabic. this shows that our method could be used to overcome arabic's complicated morphological analysis and word segmentation for all types of arabic. the code and datasets are released at https://github.com/mahmouddaif/aradic ## datasets arabic text classification lacks bench marking datasets. this is because it is expensive and time consuming to annotate a large dataset to be used for text classification using deep learning algorithms. we created two large datasets that do not require manual annotation and can be used as benchmarks for arabic text classification. the awt and the arap datasets. sections 2.1 and 2.2 describe how we constructed these datasets. ## methodology aradic is an end-to-end framework of a character encoder (ce) and a classifier. we choose two classifiers for our framework. a character cnn (clcnn) similar to @xcite , but tuned to arabic language, and a bidirectional gated recurrent unit (bigru) @xcite based classifier. the outline of our framework is shown in figure 2 . we use wildcard training introduced by @xcite for data augmentation. wildcard training is dropout on the embedding space so that the data changes a little every training iteration. in that sense it acts as a data augmentation technique. we use cb softmax loss to deal with class imbalance problem. ## experiments to train our classifier both datasets are divided into 80% training data and 20% testing data foot_1 . ## results and discussion classification results can be found in as for the classifier part of aradic, it can be noticed that the bigru significantly outperforms clcnn for both classification tasks. this suggests that sequence-to-sequence models are more suitable for text classification using image-based character-based embeddings, especially in arabic document classification. also, using cb loss improves the macro f-score of classifiers for both datasets. it can be also noted that the improvement in the macro f-score is achieved when using a clcnn and a bigru. this shows that cb loss can be useful to solve class imbalance problems for text classification tasks. figure 3 shows character embeddings visualization using t-distributed stochastic neighbor embedding (t-sne) method @xcite . as shown, embedding for related characters having similar shapes like " ", " ", " ", and " " are clustered in the embedding space. this is the majority of cases. other unrelated characters are also clustered which is rare. this however shows that using image based character embeddings gives an extra layer of visual information. another reason why it is useful is because both the ce and the classifier are trained as an end-to-end system. this means that the ce learns the best embeddings suitable for the classifier. ## conclusion in this paper, we proposed a novel end-to-end arabic text classification framework aradic. we also published two large scale arabic text classification datasets that contain the three types of arabic language, the awt and the arap datasets. aradic's image-based character embedding strategy eliminated the need for complicated preprocessing, segmentation and morphological analysis, and achieved much better performance than conventional deep and classical text classification techniques that use word and character-based embeddings. we have shown also that class-balanced loss is useful for text classification tasks with long tailed distribution datasets.