JASMINE : A rabic GPT Models for Few-Shot Learning (2023)

Abstract: Scholarship on generative pretraining (GPT) remains acutely Anglocentric, leaving serious gaps in our understanding of the whole class of autoregressive models. For example, we have little knowledge about the potential of these models and their societal impacts in diverse linguistic and cultural settings. We alleviate this issue for Arabic, a wide collection of languages and dialectal varieties with more than 400 million population, by introducing JASMINE. JASMINE is a suite of powerful Arabic autoregressive Transformer language models ranging in size between 300 million-6.7 billion parameters pretrained on a large and diverse dataset ( 235 GB of text). We also carefully design and release a comprehensive benchmark for both automated and human evaluation of Arabic autoregressive models, with coverage of potential social biases, harms, and toxicity. Using our novel benchmark, we evaluate JASMINE extensively showing powerful performance intrinsically as well as in few-shot learning on a wide range of NLP tasks. We aim to responsibly release our models and evaluation benchmark with interested researchers, along with code for experimenting with them.

## introduction recent work in generative pretraining @xcite @xcite @xcite @xcite @xcite has shown that autoregressive models perform well on language tasks using in-context learning, without finetuning or gradient updates. this in-context learning approach allows models to perform new tasks with only simple instructions and a few optional examples, which can be further improved by model adaptation through prompt tuning @xcite . in spite of this progress, autoregressive pretrained ⋆ authors contributed equally. transformer language models of significant size remain largely anglocentric. this makes it difficult to bring more diverse voices to the table. nor is it clear if multilingual models such as bloom @xcite , where model capacity is split across a large number of languages and language-specific data are neither sufficiently large nor diverse, can allow equitable understanding of these models in languages other than english. it is also not possible to study the capabilities of these models in particular linguistic environments (e.g., languages of rich morphology, of diglossic nature, and/or with a large number of dialects such as arabic) and diverse cultural backgrounds (e.g., african, asian, latin american). this situation also deprives non-english communities of the rich array of benefits language model technology can bring as its full potential and emerging capabilities @xcite are unlocked. alarmingly, we currently cannot study the social harms, risks, and biases associated with such models. in order to carefully investigate the risks of these models and work on preventing or at least mitigating them, we need to responsibly develop sufficiently large dedicated models outside english. to circumvent these limitations and advance scholarship of autoregressive models beyond english, we propose a suite of decoder-only transformer models for the arabic collection of languages and language varieties. our suite of models, dubbed jasmine, come in four different architectures that range in size from 300 million to 6.7 billion parameters. motivated by recent findings as to the impact of pretraining data size vis-à-vis model size @xcite , we carefully curate a large dataset (∼ 235gb of text) of high-quality text to pretrain jasmine. our dataset is also diverse (e.g., covers both standard and dialectal arabic), endowing our models with an ability to serve wider communities. our work also fills another significant gap for arabic autoregressive models, i.e., that of an evaluation benchmark. we introduce an evaluation benchmark comprising a wide collection of test datasets and protocols. using our benchmark, we evaluate jasmine extensively both intrinsically (using perplexity) and extrinsically (e.g., on few-shot settings). our evaluation demonstrates the superiority of jasmine compared to available baselines. we also perform human evaluations to investigate the ability of our models to write fluent and coherent standard as well as dialectal arabic across various domains (e.g., news, literary, twitter). our evaluations reveal that our jasmine models posses powerful representations, allowing them to excel in few-shot learning and produce outputs that can be identified by humans only at chance level. since autoregressive models often carry social biases, harms, and toxicity, our evaluation testbed involves the creation of a set of carefully-designed datasets for measuring a range of social risks. additionally, we aim to responsibly release our models and evaluation benchmark with interested researchers, along with code for experimenting with them. to summarize, we offer the following contributions: (1) we develop jasmine, a suite of four autoregressive language models for arabic, ranging in size between 300 million to 6.7 billion parameters pretrained with a diverse dataset. (2) we evaluate jasmine extensively, introducing a comprehensive evaluation benchmark for a wide range of nlp tasks. we demonstrate jasmine's ability to write fluent language and learn well in-context across rich contexts in few-shot settings. (3) our evaluation benchmark involves the creation and release of datasets for investigating potential social biases, harms, and toxicity. based on these evaluations, we join arms in calling for ethical practices when working with language models and inviting future research on mitigating their social risks. (4) we aim to responsibly and gradually release our models with interested researchers, along with code for experimenting with them, hoping our work will trigger applications and further research in understanding autoregressive models outside english. the rest of the paper is organized as follows: we introduce jasmine in section 2, describe our evaluation strategies in section 3, and our evaluation benchmark in section 4. in section 5, we offer human evaluations of model output. section 6 is an analysis of social bias in the model, and section 7 is about related work. we conclude in section 8. ## evaluation strategies we follow previous literature @xcite @xcite in evaluating our models extensively, under both intrinsic and extrinsic conditions as we now explain. intrinsic evaluation. perplexity (ppl) is a widely used metric that estimates how well a language model predicts a given text. for a tokenized text @xmath0, w 1 , ..., w n ), perplexity of t is: where log p 0 (w i |@xmath1. extrinsic evaluation. we employ three settings: ## evaluation benchmark we evaluate jasmine on 23 different datasets, representing five different tasks: language modeling, autocompletion, commonsense inference, word manipulation, and natural language understanding. we now introduce each of these tasks along with related datasets. ## human evaluation of model output we carry out a set of human studies to investigate the ability of our jasmine 2.7b model to generate texts from diverse domains. this includes the news, literary (i.e., poetry), and twitter domains. we also investigate the ability of the same model to produce dialectal continuations when seeded by sequences from the respective dialects. we provide sample generations from these experiments in ## analysis of social bias while autoregressive models are able to produce fluent texts which have a multitude of useful applications, they can also carry societal biases. to quantify biases in our generative models, we use conditional generation (i.e., autocomplete generation) @xcite . for all social bias experiments, we use jasmine 2.7b . we provide sample outputs from all these experiments in table e .3. biases in gender autocompletion. we investigate associations between occupation and linguistic gender by prompting the model. for this cause, we manually prepare a list of 100 occupations which we use with the following template: "@xmath2..." (e.g., ). we provide the full list in table e.1. results. we find that 62.50% of the 100 occupations we test are more likely to be followed by a male linguistic gender. this means that the model is male-leaning when an occupation context is given. gender, color, and region. inspired by kirk et al. (2021), we use the following template "you always find [x][y][z] working as . . . ", where x is a binary gender, y is one of the regions in the set {africa, asia, america, europe}, and z represents one of two colors black or white. this gives us a total of 16 prompt combinations. one example from this combination can be (english: "you'd always find black american men working as . . . "). then, we use top-k and top-p sampling (with top-@xmath3.95) to generate 10 completions for each of the 16 prompt combinations, this gives us 1, 600 generated sentences of which we keep only 1, 000 sentences that contain professions. finally, we manually classify the generated sequences into one of three categories from the manually prepared set {high-wage, medium-wage, low-wage}. ## related work large language models (llms). brown et al. (2020) develop gpt-3 and show its abilities on few-shot learning. several other works followed, usually introducing larger models @xcite @xcite . 2020 ); @xcite apply reinforcement learning to align language models for text summarization. similarly, human feedback has been used to align language models for dialogue generation @xcite , story generation @xcite , evidence extraction @xcite . most recently, @xcite use written human feedback to augment prompts and improve the performance of gpt-3. @xcite introduce sparrow, a model trained to be more helpful, correct, and harmless compared to prompted language models. instruction-tuning of llms. @xcite introduce a framework, zest, to solve a new task after reading its description. @xcite develop a novel pattern exploiting training (pet) scheme to verbalize supervised classification task into cloze question format. recently, @xcite propose instructgpt, where the authors first finetune gpt-3 with labeler-written prompts, then the authors rank the output with human feedback to align the model with the users' intent. later, chatgpt foot_16 followed the same training procedure to develop a conversational agent. taori et al. (2023 2023 ) introduce vicuna using gpt-4 (openai, 2023) to assess and rank the outputs. besides, several other models have been released based on instruction-tuning (e.g., dolly) foot_17 and rl (e.g., openassistant). 21 ethics and bias in language models. the recent success of llms is associated with various potential risks since the web pretraining datasets themselves are biased @xcite @xcite . @xcite ; @xcite show that the risk of biases gets higher with the increase of the model size, causing biases to resurface during the downstream tasks such as nli @xcite , coreference resolution @xcite , and mt @xcite . a number of ethical considerations related to plms have been studied, including memorizing and revealing private information @xcite , or spreading misinformation @xcite . ## conclusion we introduced jasmine, a suite of powerful gpt models for arabic varying in size between 300 million to 6.7 billion parameters. our models are pretrained on a large dataset of diverse arabic varieties from multiple domains. we also introduced a novel evaluation benchmark for arabic gpt models. using our benchmark, we demonstrate how it is that our models excel in few-shot learning as well as producing fluent texts that humans can only detect at chance level. we plan to responsibly release our models with researchers to support scholarship in this important research area.