A rabizi Language Models for Sentiment Analysis (2020)

Abstract: Arabizi is a written form of spoken Arabic, relying on Latin characters and digits. It is informal and does not follow any conventional rules, raising many NLP challenges. In particular, Arabizi has recently emerged as the Arabic language in online social networks, becoming of great interest for opinion mining and sentiment analysis. Unfortunately, only few Arabizi resources exist and state-of-the-art language models such as BERT do not consider Arabizi. In this work, we construct and release two datasets: (i) LAD, a corpus of 7.7M tweets written in Arabizi and (ii) SALAD, a subset of LAD, manually annotated for sentiment analysis. Then, a BERT architecture is pre-trained on LAD, in order to create and distribute an Arabizi language model called BAERT. We show that a language model (BAERT) pre-trained on a large corpus (LAD) in the same language (Arabizi) as that of the fine-tuning dataset (SALAD), outperforms a state-of-the-art multi-lingual pretrained model (multilingual BERT) on a sentiment analysis task.

## introduction nowadays, online social networks (osns) are highly popular means of communication. with pictures, videos, urls, mentions and of course comments, people share ideas, opinions and sentiments on twitter 1 , facebook 2 , linkedin 3 and many other service providers. osn posts are of great interest for marketing and reputation management, enabling companies to collect feedback about brands and products. automatic systems facilitating these large-scale tasks, including sentiment classifiers, have progressively reached promising accuracy @xcite @xcite . in parallel, the nlp field has experienced several scientific developments until the recent language models. efficient representation of texts with vectors is difficult and this challenge has seen excellent work: bag-of-words with tf*idf @xcite , word2vec @xcite , glove @xcite , fasttext @xcite , elmo @xcite , and bert @xcite for instance. each of these scientific steps has successively outperformed the previous ones on a variety of nlp tasks including sentiment analysis, with a focus on english texts. moreover, some approaches such as bert propose pre-trained language models, including a multi-lingual version. the particularity of texts post on osn is that they frequently mix made-up terms and inventive spelling with more classic syntax. twitter, for instance, contains invented words, original spelling and typing errors @xcite , but also includes special entities such as hashtags and user mentions that often appear as labels without any syntactic role @xcite . even state-of-the-art architectures such as bert usually suffer from the particularities of osn languages, as illustrated on a task of hate detection in tweets @xcite . moreover, osn posts are also the opportunity for code-switching, i.e. alternating between two or more alphabets and/or languages in conversation, in particular for many arabic speakers. as they often are familiar with latin keyboards and use them frequently, writing arabic using latin alphabet is easier and faster than switching between alphabets. this typing method, called arabizi, enables to express both modern standard arabic (msa) or arabic dialects. arabizi is a non-standard romanization of arabic script that is widely adopted for communication over the internet or for sending messages @xcite . it has emerged as the arabic language of informal communications online. arabizi appears in several challenges for computational linguistics: 1) arabizi is a written form of non-written arabic dialects; 2) there exists no broadly accepted rules or incitement to correctly write arabizi: inventive spelling is therefore a standard. this language diversity rises significant challenges to nlp: arabizi is not "official", does not follow any rules to be written and is often mixed with other foreign languages. unfortunately, only very few arabizi resources are publicly available and existing state-of-the-art language models such as bert, even in its multilingual version, are not optimised to process arabizi posts on osn. this article contributes to the study of user-generated contents in arabizi on osn, by releasing a 7.7m tweet corpus, called lad (large arabizi dataset). a subset of these tweets, named salad (sentiment annotations from lad), has been manually annotated according to their sentiment. we illustrate the interest of such resources on a sentiment analysis classification task, with an arabizi-trained bertbased language model (baert). we aim at enabling researchers to exploit lad on arabizi nlp tasks and salad to evaluate the performance of sentiment analysers on arabizi tweets; the size of salad may be insufficient to train greedy machine learning models. the remaining of this article is organised as follows. section 2 presents the characteristics of arabizi and introduces some works on sentiment analysis for arabizi. section 3 introduces our collected corpus of twitter data named lad, as well as the pre-processing and annotation tasks that lead to salad (sentiment annotations from lad). section 4 presents some experiments comparing a multilingual bert model with our arabizi baert model, pre-trained on lad and tested on salad; the results are presented in section 5. finally, section 6 concludes this article. ## arabizi, sentiment analysis and language modeling this section introduces the linguistic phenomenon of arabizi and covers the technical aspects of text classification models and their adequacy to arabizi. ## lad and salad: arabizi resources to create an open resource in arabizi and to learn an arabizi language model, we have collected arabizi tweets using egyptian keywords as seeds. the corpus is called lad (large arabizi dataset). then, we manually annotated a subset of lad according to their sentiment. this subset is named salad (sentiment annotations from lad). lad and salad are downloadable on demand foot_3 . ## results and discussion we first compare bmu-vocabulary with arz-vocabulary, then the results of the fine-tuning experiments on the two arabizi sa tasks are provided. ## conclusion and future work arabizi is a challenge for computational linguistic: there exist only few data resources, no official grammar nor orthography. moreover, the language is constantly evolving through time and space, as each country has its specific dialect. language models such as elmo @xcite or bert @xcite get state-of-the-art results on most of the nlp tasks. their pre-trained models, in english or multilingual versions, obtain excellent performance on a lot of use-cases, but do not perform correctly on arabizi, due to the lack of any large corpus in such language to carry out a pre-training step. our contribution in this article is twofold: firstly, we provided lad, a large arabizi dataset constituted of tweets. from this dataset, we made salad, a smaller manually annotated arabizi dataset for sentiment analysis, taking into account sentiment conflicts in the text or between the text and the emojis. secondly, we trained baert, a bert-like language model specific to arabizi and obtained good results on tunizi and egyptizi-like sa task, showing that a pre-training on a local specific arabizi dialect can help to get better results on this specific dialect but also on other ones. a pre-training from scratch with a new vocabulary seems preferable for a new language, even for a language that is a mix of different ones (arabizi is a mix of english, french and arabic using latin alphabet). in the future, we plan to exploit lad to train a model on different nlp tasks such as translation with english, or transliteration with msa. also, the metrics on the efficiency of the tokenizer seem an interesting lead to follow to evaluate the interest between pre-training from scratch a transformer model or exploiting pre-trained weights from an existing model. in particular, the frequence of unkown tokens seems to be promising, and need to be further investigated, with other arabizi corpora from different local dialects for example. pre-training a transformer model from scratch is computationally expensive comparing to start with already pre-trained weights, and finding metrics that can help to make this choice without any excessive computational cost could be really useful.