A ra GPT 2: Pre-Trained Transformer for A rabic Language Generation (2021)

Abstract: Recently, pre-trained transformer-based architectures have proven to be very efficient at language modeling and understanding, given that they are trained on a large enough corpus. Applications in language generation for Arabic are still lagging in comparison to other NLP advances primarily due to the lack of advanced Arabic language generation models. In this paper, we develop the first advanced Arabic language generation model, AraGPT2, trained from scratch on a large Arabic corpus of internet text and news articles. Our largest model, AraGPT2-mega, has 1.46 billion parameters, which makes it the largest Arabic language model available. The mega model was evaluated and showed success on different tasks including synthetic news generation, and zero-shot question answering. For text generation, our best model achieves a perplexity of 29.8 on held-out Wikipedia articles. A study conducted with human evaluators showed the significant success of AraGPT2-mega in generating news articles that are difficult to distinguish from articles written by humans. We thus develop and release an automatic discriminator model with a 98% percent accuracy in detecting model-generated text. The models are also publicly available, hoping to encourage new research directions and applications for Arabic NLP.

## introduction few years ago, natural language processing (nlp) was revolutionized with the introduction of multi-head self-attention transformer architecture @xcite . the transformer achieved superior performance compared to recurrent neural networks several nlp tasks including machine translation, sentence classification with bert @xcite , and electra @xcite , and sentence completion with gpt-2 @xcite , grover @xcite , and ctrl @xcite . recent works have shown that larger models pre-trained on larger datasets can further improve performance i.e. roberta @xcite , and xlm-r @xcite . on the other hand, work on arabic language modeling has mostly targeted natural language understanding (nlu) by pre-training transformerbased models using the masked language modeling (mlm) task i.e. arabert @xcite . in contrast, arabic text generation or causal language modeling hasn't received much attention. few works such as hulmona @xcite used next word prediction as a pretraining task in for transfer learning in arabic text classification. @xcite and @xcite leveraged the existing gpt2 english model and adapted it for arabic using text from the arabic wikipedia dumps, which is sub-optimal for arabic. in this paper, the first advanced language generation models built from the grounds up on arabic language have been developed. the process of pretraining aragpt2, a gpt-2 transformer model for the arabic language is described. the model comes in 4 size variants: base (135m foot_1 ), medium (370m), large (792m) and mega (1.46b foot_2 ), which allows the exploration of aragpt2 in multiple applications with different data availability and computational constraints. the perplexity measure is used to automatically evaluate aragpt2. furthermore, a human-based evaluation is provided, which highlights the ability of aragpt2 to deceive human evaluators. finally, an araelec-tra @xcite based detector is devel-oped and released. it is able to consistently identify news articles written by aragpt2. making such powerful models publicly available to the arabic research community enables research in rising arabic nlp fields i.e conversational agents @xcite , detection of automatic news generation detection @xcite ... our contributions can be summarized as follows: ## aragpt2: methodology aragpt2 is a stacked transformer-decoder model trained using the causal language modeling objective. the model is trained on 77gb of arabic text. aragpt2 comes in four variants as detailed in ## conclusion aragpt2 is the first advanced arabic language generation model based on the transformer architecture. the model was trained on the largest publicly available collection of filtered arabic corpora. the model was evaluated using the perplexity measure which measures how well a probability model predicts a sample. results show that aragpt2 is able to produce high quality arabic text that is coherent, grammatically correct and syntactically sound. it is important to note that aragpt2, like many ml models, has ethical implications and can be used maliciously i.e. automatic fake news generation, modeling the dataset inherent biases... to help detect misuse of the model, a detector model that is tasked to detect output generated by aragpt2 is also released. more importantly, our hopes that publicly releasing aragpt2 will open up doors for new research possibilities for the arabic nlp community.