A rabic T ransformer: Efficient Large A rabic Language Model with Funnel Transformer and ELECTRA Objective (2021)

Abstract: Pre-training Transformer-based models such as BERT and ELECTRA on a collection of Arabic corpora, demonstrated by both AraBERT and AraELECTRA, shows an impressive result on downstream tasks. However, pre-training Transformer-based language models is computationally expensive, especially for large-scale models. Recently, Funnel Transformer has addressed the sequential redundancy inside Transformer architecture by compressing the sequence of hidden states, leading to a significant reduction in the pre-training cost. This paper empirically studies the performance and efficiency of building an Arabic language model with Funnel Transformer and ELECTRA objective. We find that our model achieves state-of-the-art results on several Arabic downstream tasks despite using less computational resources compared to other BERT-based models.

## introduction the introduction of transformer and attention mechanism @xcite have achieved significant success by exploiting transfer learning. bidirectional encoder representations from transformers bert @xcite , builds upon the idea of pre-training a transformer with selfattention on large amounts of unlabeled text. then, leverage the idea of transfer learning to fine-tune the pre-trained language model on downstream tasks. bert has achieved impressive performance gains against its predecessor bi-lstm @xcite on many downstream tasks. in the arabic domain, both @xcite and @xcite have adapted bert and electra @xcite models to the arabic language and show impressive results on downstream tasks. however, pre-training transformer-based models, especially at a large scale, requires enormous computational resources. this issue motivates us to investigate a solution to reduce the cost of pretraining transformer-based models. reducing the cost to train arabic language models will help accelerate research advancement in arabic language processing. additionally, this will help researchers with limited resources to fine-tune large models. several techniques in the literature have suggested solutions to reduce the cost of pre-training and fine-tuning, including cross-layer parameter sharing with albert @xcite and distillation @xcite . distillation and similar techniques have a detrimental effect on performance since they aim to reduce the parameter size. on the other hand, the fine-tuning and inference time for the albert model, especially for albert xlarge and albert xxlarge scale is significantly higher than bert large and electra large as a result of having more hidden layer size. thus, we seek alternative architectures that could increase the scale of the model without adding additional cost to the pre-training. funnel transformer @xcite introduces a novel solution to address the cost of pre-training by reconstructing the transformer architecture using pooling and up-sampling techniques. additionally, electra speeds up the pre-training by introducing a new objective function, employing a small generator model trained with maximum likelihood. this study investigates the effect of pretraining funnel transformer with electra objective on the performance of arabic downstream tasks. our results show that we achieve state-ofthe-art results with less computational resources than existing arabic language models described in the literature. thus, our contributions in this paper include : figure 1: overview of funnel transformer architecture. figure adapted from @xcite . ## fine-tuning on downstream tasks 4.1 question answering to compare our model with existing models in the literature, we use arcd @xcite and the arabic portion of tydi qa @xcite . both arcd and tydi qa are in format of squadv1.1 dataset @xcite . similar to the araelectra and arabert approach, we fine-tune our model on both arabicsquad and arcd training datasets. then, we evaluate our model on the test portion of the arcd dataset. moreover, as is a common practice, we use a preprocessing script developed by the aub mind lab, which fixes the position of text spans and handles special characters in the arcd dataset. our baseline models for qa tasks including arabertv02 large , arabertv2 large (antoun et al., 2020), arabic-albert xlarge @xcite and araelectra. we follow the same split of training and development dataset used by araelec-tra, summarized in ## conclusion we introduce arabic transformer, a pretrained arabic language representation model based on funnel transformer and electra objective. we show that we achieve state-of-the-art results on several arabic downstream tasks, including question answering and sentiment analysis tasks. additionally, we show that our models are computationally efficient and pretrained using significantly less resources than state-of-the-art models. for future work, we plan to investigate different designs of the funnel transformer, including larger models such as (b8-8-8).