A l G hafa Evaluation Benchmark for A rabic Language Models (2023)

Abstract: Recent advances in the space of Arabic large language models have opened up a wealth of potential practical applications. From optimal training strategies, large scale data acquisition and continuously increasing NLP resources, the Arabic LLM landscape has improved in a very short span of time, despite being plagued by training data scarcity and limited evaluation resources compared to English. In line with contributing towards this ever-growing field, we introduce AlGhafa, a new multiple-choice evaluation benchmark for Arabic LLMs. For showcasing purposes, we train a new suite of models, including a 14 billion parameter model, the largest monolingual Arabic decoder-only model to date. We use a collection of publicly available datasets, as well as a newly introduced HandMade dataset consisting of 8 billion tokens. Finally, we explore the quantitative and qualitative toxicity of several Arabic models, comparing our models to existing public Arabic LLMs.

## introduction recent advances in the field of ai, and particularly the development of large language models (llms), have been driven by a convergence of factors including the availability of large amounts of unlabelled textual data @xcite , advancements in hardware @xcite , software @xcite , compute infrastructure @xcite , as well as algorithmic innovations @xcite . without doubt, all these factors combined have accelerated the progress and capabilities of ai, leading to the emergence of large language models @xcite . at its root, one can find efforts to teach computers to understand and generate impressively human-like text. these efforts began with relatively simple statistical models @xcite and rule-based systems, but in recent years, the field has been revolutionized by the advent of deep learning and the availability of large-scale computational resources and data @xcite . the inaugural iteration of generative pretrained transformer (gpt) @xcite demonstrated the efficacy of causal language modelling as a pre-training objective, where the model is trained, auto-regressively, to learn the probability of a word given previous context, substantively enhancing the model's ability for generalization. subsequently, gpt-2 @xcite provided empirical evidence that augmenting both the size of the model and the volume of the training dataset enables surpassing previously established benchmarks in numerous tasks within a zero-shot framework. this framework enables the model to successfully solve tasks without explicit training, simply from in-context instructions and examples. the strategy of scaling gpt models was taken to its zenith with the introduction of @xcite , a model comprising an unparalleled 175-billion parameters. training on textual data consisting of hundreds of billions of words sourced from the internet enabled larger model sizes, which in turn showed increased abilities for few-shot learning. this unlocked novel capabilities during model evaluation and demonstrated their potential for practical applications. in recent years, a series of large language models (llms) have been introduced: gopher @xcite , palm @xcite , llama2 @xcite , with the largest dense language models now having over 500 billion parameters. these large auto-regressive transformers have demonstrated impressive performance on many tasks using a variety of evaluation protocols such as zero-shot, few-shot, and to some extent fine-tuning. further research revealed that larger models systematically deliver better language modelling performance @xcite , retaining more complex relationships and more subtleties of the language. larger models were shown to also capture more contextual information than smaller models, demonstrating improved emergent downstream capabilities @xcite . however, given the substantial increase in compute needs and the potential energy cost considerations associated with the training of such large language models @xcite , several works have gone into discovering the optimal allocation between the number of model parameters and data samples used. this has led to the formalism of power law scaling relationships between the number of model parameters and training tokens, given a computational budget @xcite . recent results regarding the scaling of these model @xcite have confirmed that model performance is linked with the availability of large, high-quality @xcite , and diverse datasets. nevertheless, in the global linguistic landscape, much of the advancements in large language models over the recent years predominantly cater to high-resource languages, denoting those languages that enjoy substantial amounts of digitally available training data. here english stands at a privilege, still covering ∼ 46% of recent common-crawl dumps, followed at 4 -6% each by german, russian, french, japanese, spanish, and chinese foot_0 . these languages stand to profit massively from the progression of language models in contrast to a significant proportion of languages, often characterized by their lower resources, and which attract less attention, despite their cumulative prevalence foot_1 . here, arabic represents a case of particular note, as it is the native tongue of 360 million people (including dialects) and the official language of 27 states and territories, but its overall presence on common-crawl for example is ∼ 0.5% (∼ 0.66% in recent dumps ). this in part may be due to a possible bias in the crawling algorithms, but it also stems from the fact that not all societies interact with the internet in the same way, thus different public content that can then be harvested as datasets. the main contributions of the present work are: ## related work in the past three years, several arabic generative language models have been published (with a few being publicly available), exploring different architectures (bert, gpt and t5-based) and increasing model sizes, while facing limitations in training data and evaluation resources. aragpt2 @xcite was the first dedicated arabic generative language model to be developed where the training corpus included arabic data from internet and news articles. the largest model in this family, aragpt2-mega, with 1.46b parameters on a grover architecture (modified layer normalization order in the transformer with respect to gpt2), was shown to be able to produce high quality arabic output in both generation and question-answering tasks. a larger gpt-based arabic model, was introduced by @xcite . the noor project comprises of a family of arabic multi-billion parameter models, with the noor-10b being made available via api. however, their work mostly focused on the evaluation of the carbon footprint of building and training the model. nagoudi et al. (2022) introduced a range of gpt models (300m to 13b parameters), trained on 400 gb of text, with the largest model (jasmine-13b) still in training at the time of publication. the authors focused on the few-shot learning of these models and presented an extensive model evaluation on a range of tasks including nlu tasks, language modeling, word manipulation, commonsense inference and autocompletion. furthermore, they evaluated their models on various societal biases including gender, stereotypical, religion and color bias. in line with evaluating the capabilities of arabic llm, @xcite recently released jais and jais-chat. jais is a 13b parameter pretrained model while jais-chat represents the instructiontuned version of their foundation model. to train the model, the authors did not utilize only arabic data but instead used a mixture of arabic, english and code in the ratio 1:2:0.4. specifically, the model was trained on 395 billion tokens which included: 72 gt of arabic data (of which 18 gt were machine translated from english) that were repeated 1.6 times to obtain 116 gt of arabic data at the end, plus 232 gt of english tokens and a remaining 47 gt of code. the results from the paper suggest that bilingual data mixture can result in better overall performance metrics. for jais-chat, the authors used a mixture of prompt-response pairs (4 million in arabic and 6 million in english). in the space of bert-based models, @xcite posit that existing arabic models are largely under-trained which affects their performance significantly. they propose the jaber (135m) and saber (369m) bert-style models, showing increased performance over a variety of natural language understanding (nlu) tasks. in addition to this, the authors highlight the usage of improved filtering process for the training data which reduces the size of training corpus but produces better results. following this strategy, alghamdi et al. (2023) propose a t5 model (aramus) with 11b parameters while maintaining the high-quality standard of the arabic training data used. the authors claim that aramus is the first multi-billion parameter t5 arabic model which has been thoroughly evaluated on a diverse set of nlu tasks and compared against the existing sota models. its performance, evaluated on the alue benchmark (seelawi et al., 2021) present state-of-the-art results among bert and t5 models. parallely, nagoudi et al. (2021) introduced arat5 for transfer learning in arabic and pretrained three models, one trained on modern standard arabic (msa), another one on twitter data and last on both msa and twitter. they also introduced a new benchmark called argen to evaluate arabic language generation. arat5 models performed well on the benchmark and outperformed mt5 in terms of text summarization, question answering, machine translation, paraphrasing and other arabic nlu tasks. ## model a de facto architecture for large language models, the canonical transformer architecture @xcite , has seen several improvements to enhance the overall model qualitative performance and speed up both training and inference workloads. our family of arabic models are a suite of decoder based generative models @xcite , closely following the architecture of the falcon models foot_3 which in turn was modified from the gpt-3 architecture @xcite . we highlight the following attributes: more details on model architecture are given in ## toxicity and bias analysis we address the study of stereotypical bias related to gender, religion and ethnicity following two distinct approaches, respectively a descriptive and a quantitative one. ## limitations as our models are trained chiefly on publicly available arabic data crawled from the internet (∼ 95%) and cleaned using a large-scale automated pipeline, they can present to some degree several of the issues commonly found in large language models: outputting incorrect/private/sensitive information, toxicity and/or bias, the potential for misuse. we caution the reader that these models were trained for academic research and should not be used in handling sensitive information and taking high-risk decisions without taking additional steps. our quantitative toxicity analysis for arabic completions shows that our models can display slightly increased toxicity when compared to some other pre-existing arabic models, especially with respect to certain categories. we show this can be significantly alleviated through fine-tuning. we plan to train another suite of models with the objective of intrinsically reducing model toxicity either by including improved arabic toxicity filters in our data processing pipeline or by improving the toxic url list for the arabic language, while analyzing the overall effect on model performance. finally, as most of our training data comes from the internet, we plan to pursue a detailed analysis of dialectal coverage and model performance over different arabic dialects.