A rabic Compact Language Modelling for Resource Limited Devices (2021)

Abstract: Natural language modelling has gained a lot of interest recently. The current state-of-the-art results are achieved by first training a very large language model and then fine-tuning it on multiple tasks. However, there is little work on smaller more compact language models for resource-limited devices or applications. Not to mention, how to efficiently train such models for a low-resource language like Arabic. In this paper, we investigate how such models can be trained in a compact way for Arabic. We also show how distillation and quantization can be applied to create even smaller models. Our experiments show that our largest model which is 2x smaller than the baseline can achieve better results on multiple tasks with 2x less data for pretraining.

## introduction language modelling refers to the process of designing a model that can mimic language understanding in humans. the trained models can then be finetuned on multiple tasks by using them as feature extractors by adding extra layers at thend of such models. this is a difficult task since, language understanding and comprehension require huge amounts of data. recently, advances in machine learning and particularly deep learning is proving significant in different tasks like text classification, sentiment analysis, question answering, summarization and language inference using transformer-based architectures @xcite . this approach requires training a deep architecture that models the language using unsupervised techniques on unlabelled data. the outcome of this method is a large model that has a huge memory footprint. this limits the deployment of such models on resourcelimited devices because of memory constraints. furthermore, the model needs to be retrained to make it useful for other more interesting tasks. re-cently, a lot of approaches have emerged in making these deep architectures more practical using methods like distillation @xcite and quantization @xcite . unfortunately, most of the work that is available in the literature targets english language because of the availability of data and research development tools. in this work we train a compact arabic language model that can be fine-tuned on different tasks. moreover, we make the model efficient for embedded devices with relatively a small loss in accuracy. the paper is organized as follows: in section 2, we revise the literature for related work. in section 3, we describe our methodology where we discuss the pretraining dataset, the tasks used for evaluation and our training procedure for the teacher and student models. finally, we summarize the experiments and discuss the results. ## related work deep architectures like gpt @xcite , bert @xcite and t5 @xcite require a huge amount of data and computation. moreover, such language models are very large and cannot be deployed on embedded devices due to memory constraints. recently, many approaches have been applied in order to make such models smaller and more practical. mainly, the literature focuses on two main approaches which are quantization and distillation. quantization is defined as reducing the memory footprint by mapping parameters to low dimensional spaces. for instance, we can map float32 to int16. this is usually done at the deployment phase where the model is already trained. many studies have been published that adopt this approach. zafrir et al. applied this approach to bert in order to create a quantized 8 bit model (q8bert) @xcite . more recently, a more advanced approach called hessian-based ultra low precision quantization has been applied to bert (q-bert) @xcite . in another work, the authors designed an architecture called gobo that targets quantizing attention-based models for language understanding @xcite . on the other hand, another approach is called distillation which mimics the task of a teacher and student for training smaller language models (alkhulaifi et al., 2020). these methods have been developed in computer vision but they were recently extended to language modelling. the teacher model, usually a large model, is a pretrained model that can be used to distil the knowledge to a smaller model called the student model. the student model is a much smaller architecture. the distillation process has been applied in many fields in machine learning and takes many forms like teacherstudent optimization @xcite , teacher assistant @xcite , contrastive representation distillation @xcite and patient knowledge distillation @xcite . other approaches make the process task-agnostic like the work by sun et al. in their architecture called mo-bilebert @xcite . from the research perspective, arabic language modelling is under-represented in the literature let alone implementing compression techniques. noticeably, there were a few main attempts to adapt deep models for arabic like hulmuna @xcite which retrains elmo @xcite for arabic and fine-tunes it on different tasks. moreover, we have arabert which retrains bert for arabic @xcite . more recently, other larger variants have been introduced namely @xcite . these models are comparatively very large and are trained with a huge amount of data usually larger than 20 gb of text and the models are typically more than 100 million parameters in size. arabic poses its own challenges when considered for natural language processing tasks. arabic is a morphologically rich language where words can take different representations by connecting different prefixes, infixes and suffixes. hence words need better representations in order to tackle the out-of-vocabulary problem @xcite by applying different preprocessing approaches. for instance, segmentation slightly achieves a better performance compared to direct tokenization @xcite . not to mention applying prepro-cessing to deal the linguistic parts like diacritics, normalization, cleaning, mixed foreign characters, etc. in this work we consider applying distillation and quantization on arabic language models for resource-limited devices. up to our knowledge, there is no previous work that applies compression directly on arabic. there are some related work that applies distillation on a cross-lingual level @xcite but it is evident from the literature that monolingual models are more powerful @xcite , @xcite . ## experimentation in this section we explain in details our approach that we take for training compact language models for arabic. first we train a large teacher model in an unsupervised fashion on a large corpus. then we distil the knowledge from the teacher model to the student model and evaluate it by fine-tuning on different tasks. finally, we use quantization as a final step for reducing the memory footprint of our distilled student models. for the comparison we use the first version of arabert as the baseline for evaluating our models. ## evaluation we use fine-tuning on down-stream tasks as the main approach for evaluating our models. in all of our experiments, we follow the same procedure for the teacher and the student models. note that in order to make the comparison fair to arabert, we don't do any preprocessing to our datasets. this is in order to avoid any bias towards our models. this might cause our models to preform worse than expected because some characters in the datasets might cause unknown tokens to show up. mainly, we train for 5 epochs and we use a learning rate of 3 × 10 -5 for the text classification and question answering tasks while we use a learning rate of 2 × 10 -3 for the pos tagging task. in figure 1 , we compare inverted-block bert for arabic (iberta) for two different vocabulary sizes. to make the comparison fair, we decrease the hidden size to 256 for the model with vocabulary size 50k. we see as the number of iterations increases, the accuracy increases for both models. however, the model with vocabulary size 50k achieve better results across the different iterations compared to the vocabulary size 30k. this is due to the fact that arabic is morphologically rich language with a huge and sparse vocabulary. we use the model with the vocabulary size 50k as the teacher model and we refer to it as (iberta). in ## conclusion in this paper we trained compact arabic language models through pretraining, distillation and quantization. we showed that such compact language models can achieve comparable results to state-ofthe-art models even though they have significantly a fewer number of parameters and while trained on smaller datasets. we benchmark the models on six different classification datasets, one part-ofspeech tagging dataset and one question answering dataset. we believe such smaller models can be very practical because they enable deploying such models on resource-limited devices like mobile phones and embedded devices. moreover, we showed that using smaller datasets with smaller vocabulary, we can achieve reasonable results compared to other models in the literature. this shows that carefully curated datasets with high diversity even if smaller can get some good results. we believe that we can achieve better results by applying some hyper-parameter tuning through approaches like grid search. it is also interesting to look at how the parameters of mobilebert distillation affect the training process and if changing some of them from the original model can achieve better results for arabic. also, a closer look at the distillation process can give us some estimate about the bias towards a certain linguistic phenomenon @xcite .