Benchmarking Transformer-based Language Models for A rabic Sentiment and Sarcasm Detection (2021)

Abstract: The introduction of transformer-based language models has been a revolutionary step for natural language processing (NLP) research. These models, such as BERT, GPT and ELECTRA, led to state-of-the-art performance in many NLP tasks. Most of these models were initially developed for English and other languages followed later. Recently, several Arabic-specific models started emerging. However, there are limited direct comparisons between these models. In this paper, we evaluate the performance of 24 of these models on Arabic sentiment and sarcasm detection. Our results show that the models achieving the best performance are those that are trained on only Arabic data, including dialectal Arabic, and use a larger number of parameters, such as the recently released MARBERT. However, we noticed that AraELECTRA is one of the top performing models while being much more efficient in its computational cost. Finally, the experiments on AraGPT2 variants showed low performance compared to BERT models, which indicates that it might not be suitable for classification tasks.

## introduction in recent years, the development of contextualised language representations led to a revolution in the natural language process (nlp) field. early work on representing language started with pretrained word representations such as word2vec @xcite , glove @xcite and fasttext @xcite . however, these word representations were static and did not rely on the context, in which they appear. other works tried to address this issue with contextualised word representations such as elmo @xcite . @xcite proposed ulmfit, which is a method for transferlearning that can be applied to any task in nlp. the introduction of bert @xcite revolutionised the research on nlp. bert is based on complicated neural network models, namely transformers. the utilisation of bert led to achieving state-of-the-art results on many nlp tasks such as sentence completion, question answering and sentiment analysis. the advantage of bert and similar models is that they are pre-trained on large amounts of data, which the model utilises to learn a representation of the language. following bert, many other transformer-based language models were released such as electra @xcite , gpt-1/2/3 @xcite and t5 @xcite . these models tried to improve the performance of bert through some slight modifications on the training objective such as roberta @xcite . or completely changing the training methodology such as electra @xcite . these models were mostly specific for english with the exception of some multilingual ones such as the multilingual bert (mbert) @xcite . researchers on other languages followed the trend and released other language-specific models such as camambert @xcite for french, phobert (nguyen and tuan nguyen, 2020) for vienamese, finbert @xcite for finnish, bertje @xcite for dutch and others. arabert @xcite was the first arabic-specific transformer-based language model. the introduction of arabert helped improving the performance in many arabic nlp tasks. recently, a large set of transformer-based arabic language models has been released. these include bert based models such as the new large version of @xcite , qarib @xcite , arbert/marbert @xcite . also, arabic variants of other models were released such as aragpt2 @xcite , araelectra @xcite and arabic albert (kuis-ai-lab). these models vary in their architectures, sizes and the nature of their training data. while most of these models were trained on modern standard arabic (msa) data; some of them, such as marbert, included dialectal arabic in their training data. most of these models were evaluated on a small set of arabic nlp tasks and without any direct comparison with other models. thus, there is no clear measure of the effectiveness of one of these models on a specific task compared to the others. hence, in this paper, we aim to provide a comparative study of the performance of all of the recently introduced arabic language models for the wellstudied arabic sentiment analysis (sa) task and the emerging arabic sarcasm detection task. in this paper, we provide a rigorous comparison of the effectiveness of 24 recently-released arabic language models. these models were evaluated for the tasks of arabic sentiment analysis and sarcasm detection. we test these models on the newly released arsarcasm-v2 dataset @xcite , which was released along with the shared task on arabic sarcasm detection. the experiments show the following: first, the models trained on dialectal arabic are the most effective to handle the tasks under study, where the best model marbert achieved an f p n of 0.724 on the sa task and f1-sarcastic of 0.584 on the sarcasm detection task. second, language-specific models achieve higher results than the multilingual ones. third, the training procedure of electra is better than the other models, as araelectra @xcite achieved results close to larger models while being smaller and more efficient. fourth, aragpt2 is not suited for classifications tasks as it performed worse than all the other models. ## summary of findings from the experimentation of the existing transformer-based lms on arabic sentiment and sarcasm detection, we can learn the following: in the experiments, arabic-specific models achieved higher results than their multilingual equivalents. larger multilingual models achieve similar scores to smaller monolinguals. we hope that this benchmark study would guide future research in arabic sentiment and sarcasm detection, and generally in arabic nlp tasks including the creation of new language resources.