A ra ELECTRA : Pre-Training Text Discriminators for A rabic Language Understanding (2021)

Abstract: Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.

## introduction recently, pre-trained language representation models have demonstrated state-of-the-art performance on multiple nlp tasks and in different languages. pre-training is commonly done via masked language modeling (mlm) @xcite @xcite , where an input sequence has some of its tokens randomly hidden and the model is tasked to recover the original masked tokens. while this approach has proven successful, recent works have shown that mlm is not sample-efficient @xcite , since the network only learns from the small subset of masked tokens per sequence (15% of the tokens in bert). @xcite proposed an approach called efficiently learning an encoder that classi-fies token replacements accurately (electra). the method uses a pre-training technique based on replaced token detection (rtd) task is more efficient than mlm, and thus achieved state-of-theart results on english benchmarks. rtd is a pretraining task where a model is tasked to distinguish true input tokens from synthetically generated ones. rtd solves the issue of the mismatch created in mlm, where the model only sees the [mask] token during pre-training but not during fine-tuning. in electra, a small masked language generator network g is used to generate used to generate the corrupted tokens, and bert-based discriminator model d predicts for whether a token is an original or a replacement. current state-of-the-art language representation models for arabic employ mlm as a pre-training objective @xcite @xcite @xcite . in this paper, we describe the process of pre-training a transformer encoder model for arabic language understanding using the rtd objective, which we call araelectra. we also evaluate araelectra on multiple arabic nlp tasks and show empirically that araelectra outperforms current state-of-the-art arabic pre-trained models. our contributions can be summarized as follows: the rest of the paper is organized as follows. section 2 provides a review of previous arabic language representation literature. section 3 details the methodology used in developing araelec-tra. section 4 describes the experimental setup, evaluation procedures, and experiment results. finally, we conclude in section 5. ## related works recently, work on arabic language representation have been on the rise due to the performance benefits that transfer learning approaches have brought. early transfer learning approaches in arabic relied on using pre-trained word embeddings i.e. ar-avec @xcite . model-level transfer learning was shown to work on arabic with hulmona @xcite , a recurrent neural network-based language modeling approach. antoun et al. (2020) and safaya et al. (2020) improved on hulmona, and pre-trained transformerbased models with mlm with large scale arabic corpora. other approaches addressed issues with the early bert-based models such as training on code-switched english-arabic corpora to improve performance on information retrieval tasks @xcite , and training on dialectal arabic (da) corpora to address the domain miss-match between msa and da during pre-training and finetuning @xcite . we hence propose an arabic electra-based language representation model pre-trained using the rtd objective on large msa corpora. ## araelectra: methodology in this paper, we develop an electra-based arabic language representation model to improve the state-of-the-art in arabic reading comprehension. we create araelectra a bidirectional transformer encoder model with 12 encoder layers, 12 attention heads, 768 hidden size, and 512 maximum input sequence length for a total of 136m parameters. the pre-training setup and dataset of araelectra are described in the following sections. ## conclusion in this paper, we showed that pre-training using the rtd objective on arabic text is more efficient and produces pre-trained language representation models better than the mlm objective. our araelectra model improves the stateof-the-art for arabic question answering, senti-ment analysis and named-entity recognition, and achieves higher performance compared to other models pre-trained with the same dataset and with larger model sizes. our model will be publicly available, along with our pre-training and finetuning code, in our repository github.com/aubmind/arabert/tree/master/araelectra