Improving A rabic Text Categorization Using Transformer Training Diversification (2020)

Abstract: Automatic categorization of short texts, such as news headlines and social media posts, has many applications ranging from content analysis to recommendation systems. In this paper, we use such text categorization i.e., labeling the social media posts to categories like ‘sports’, ‘politics’, ‘human-rights’ among others, to showcase the efficacy of models across different sources and varieties of Arabic. In doing so, we show that diversifying the training data, whether by using diverse training data for the specific task (an increase of 21% macro F1) or using diverse data to pre-train a BERT model (26% macro F1), leads to overall improvements in classification effectiveness. In our work, we also introduce two new Arabic text categorization datasets, where the first is composed of social media posts from a popular Arabic news channel that cover Twitter, Facebook, and YouTube, and the second is composed of tweets from popular Arabic accounts. The posts in the former are nearly exclusively authored in modern standard Arabic (MSA), while the tweets in the latter contain both MSA and dialectal Arabic.

## related studies the task of categorizing social media posts is challenging mainly due to the absence of largely annotated data. most of the datasets that are currently available are based on news articles. some publicly available datasets are: 1. sanad foot_1 (alsaleh et al., 2020): this is one of the largest collection of news article for arabic news text classification. this multi-class dataset includes news articles that are scraped from the alkhaleej, foot_2 alarabiya, foot_3 and akhbarona foot_4 news portals. the dataset includes approximately 195k articles belonging to 6/7 categories. 2. nadia foot_5 (elnagar et al., 2020): this is a multi-label text dataset, collected by scraping skynewsarabia foot_6 and masrawy foot_7 news sites. the released dataset include 486k articles with 52 categories. 3. arabic news text (ant) corpus foot_8 (chouigui et al., 2017): this dataset was collected from rss feeds and contains approximately 6k articles belonging to 9 categories other available datasets include: khaleej-2004 (5k articles belonging to 4 categories) @xcite ; watan-2004 (20k articles belonging to 6 categories) @xcite ; and sl-rtanew foot_9 (20k articles belonging to 40 categories). to capture syntactic and semantic information about words, pre-trained word static embeddings @xcite @xcite were widely used in many nlp tasks. recent research advancements led to pre-trained contextual embeddings that capture much information about words in context, leading to significant improvements for many nlp tasks such as text classification and sequence labeling @xcite @xcite @xcite @xcite . as for arabic, various static and contextual embeddings representation have been trained. some popular arabic static embeddings include: arabic word2vec @xcite , a fasttext model that is trained on wikipedia @xcite , and dialectal word embeddings with small and noisy corpora @xcite and with tweets @xcite . as for contextual embeddings, a handful of models are available @xcite @xcite . the first available bert model for arabic was multilingual bert (mbert), which was pre-trained on the wikipedia dumps of 104 languages including arabic. however, previous studies have shown that monolingual bert models perform significantly better than the mbert @xcite . a recent arabic bert model (arabert) (antoun et al., 2020) was trained on wikipedia and a large collection of arabic news articles, with the base configuration of the bert model. the model showed success for many arabic nlp downstream tasks. recently, a multidialect-arabic-bert @xcite model was released and entailed fine-tuning arabert on 10m tweets. the model was used to improve dialect identification. compared to the available datasets, which are mainly based on news articles, our introduced datasets are based on social media platforms. the datasets include posts written in standard and dialectal arabic from multiple social media platforms, and labeled with 12 news categories. we publicly released the datasets for the research community. unlike most of the previous studies, we empirically show the effectiveness of monolingual bert in compare to multilingual bert for arabic language processing and the importance of having diversely (pre-)trained bert model for social media post classification task. ## datasets in this paper, we used three different datasets. two of them are written in msa and contain short social media posts and news headlines, and the third is composed of tweets, many of which are authored in dialectal arabic. ## results and discussion mismatch in style leads to lower results. though both sanad and asnd use formal text, training on one and testing on the other produces significantly lower results compared to training and testing on sanad alone. we suspect that this due to the difference in style between news headlines and social media posts, where the latter contain platform specific features such as hashtags and mentions. the drop in the effectiveness may indicate the inability of the models to generalize well when the style changes. combining dataset of different styles helps. as the results show, training using asnd+sanad performed on sanad at par to training on sanad. further combining both training sets led to improved results for all models when testing on aitd. the results of training on both and testing on asnd yielded mixed results, where results of using bert models improved significantly or matched the results of training on asnd alone. this was not the case for svm, where the results dropped noticeably. a bert model trained on a mixture of formal and informal data has much better generalization power compared to bert models that are trained on formal text only. this observation is apparent across all experiments where we conducted cross-dataset training and testing and there was a mismatch in style or language variety between them. for example, when training on asnd and testing on sanad, qarib results were 21 points better than using arabert (f1 of 81 compared to 60). a similar result is observed when training and testing on social media posts. when training on asnd or asnd+sanad and testing on asnd, qarib led to significant improvements over mbert and arabert. results of testing on aitd show the same trend. this indicates the importance of a trained bert model with mixed style data for effective domain adaptation. to further understand the impact of having pre-existing style/domain knowledge, we analyzed the difference in per class predictions. from the figure 1 , we observe that qarib performed significantly better for the majority of the classes. figure 1 : class-wise f-measure performance on asnd test set using mbert:asnd+sanad, arabert:asnd+sanad and qarib:asnd+sanad models. ## conclusion in this paper, we investigated the effect of pre-training a bert model on a mixture of formal and informal text on text categorization compared to bert models that were trained exclusively on formal text. we show that the former has greater generalization power, compared to the latter, and is able to significantly classify texts from different sources, such as news headlines and social media posts, and different varieties of arabic, namely msa and dialectal arabic. we also introduced two new arabic multi-class short text datasets. the first contains social media posts from the official twitter, facebook, and youtube accounts for aljazeera news channel. though they are social media posts, they are written in msa. the second dataset contains tweets from popular twitter accounts, with large portion of their tweets being authored in dialectal arabic. the key observation in our work is that diversifying the training data, whether by using diverse training data for a specific task or using diverse data to pre-train a bert model, leads to overall improvements in classification effectiveness.