A ra T 5: Text-to-Text Transformers for A rabic Language Generation (2022)

Abstract: Transfer learning with a unified Transformer framework (T5) that converts all language problems into a text-to-text format was recently proposed as a simple and effective transfer learning approach. Although a multilingual version of the T5 model (mT5) was also introduced, it is not clear how well it can fare on non-English tasks involving diverse data. To investigate this question, we apply mT5 on a language with a wide variety of dialects–Arabic. For evaluation, we introduce a novel benchmark for ARabic language GENeration (ARGEN), covering seven important tasks. For model comparison, we pre-train three powerful Arabic T5-style models and evaluate them on ARGEN. Although pre-trained with ~49 less data, our new models perform significantly better than mT5 on all ARGEN tasks (in 52 out of 59 test sets) and set several new SOTAs. Our models also establish new SOTA on the recently-proposed, large Arabic language understanding evaluation benchmark ARLUE (Abdul-Mageed et al., 2021). Our new models are publicly available. We also link to ARGEN datasets through our repository: https://github.com/UBC-NLP/araT5 .

## introduction due to their remarkable ability to transfer knowledge from unlabeled data to downstream tasks, pre-trained transformer-based language models have emerged as important components of modern natural language processing (nlp) systems. in particular, the unified framework that converts all text-based language problems into a text-to-text format presented through the t5 model @xcite is attractive. in addition to its simplicity, this approach is effective since it allows knowledge transfer from high-resource to low-resource tasks without the need for changing model architecture. unlike models such as bert @xcite , which are based on encoders only, the t5 model is an encoder-decoder that can naturally be employed for natural language generation. although the t5 model, originally pre-trained for english, was recently extended to the multilingual setting as mt5 @xcite , it is not clear how suited it is to individual languages (and varieties of these languages). in addition, systematic issues have been discovered in multilingual corpora on which language models have been trained @xcite . in absence of comparisons with monolingual pre-trained language models that serve different non-english contexts, it remains unknown how multilingual models really fare against languagespecific models. in this work, we offer the first comparison of the mt5 model to similar encoder-decoder models dedicated to arabic. we choose arabic as our context due to its large set of diverse varieties as well as its wide use on social media. our work aims at uncovering the extent to which mt5 can serve arabic's different varieties. our work also meets an existing need for pre-trained transformer-based sequenceto-sequence models. in other words, while several bert-based models have been pre-trained for arabic @xcite @xcite , no such attempts have been made to create sequence-to-sequence models that we know of. another motivation for our work is absence of an evaluation benchmark for arabic language generation tasks. apart from machine translation where researchers are starting to propose benchmarks such as arabench @xcite , there are no benchmarks that can be used to methodically measure arabic natural language generation performance. our main contributions are as follows: the rest of the paper is organized as follows: section 2 describes our arabic pre-tained models. in section 3, we introduce argen, our new natural language generation benchmark. we evaluate our models on argen in section 4. section 5 is an analysis and discussion of our results. in section 6, we provide an overview of related work. we conclude in section 7. we now introduce our new pre-trained models. we use 70gb of msa text (7.1b tokens) from the following sources: aranews (nagoudi et al., 2020), el-khair el-khair (2016), gigaword, foot_1 , oscar (suárez et al., 2019), osian @xcite , wikipedia arabic, and hindawi books. 3 twitter data. we randomly sample 1.5b arabic tweets (178gb) from a large in-house dataset of ∼ 10b tweets. we use string matching to only include tweets with at least 3 arabic words, regardless whether the tweet has non-arabic string or not. our combined msa and twitter data make up 29b tokens, and hence is ∼ 49% less than arabic tokens on which mt5 is pre-trained (57b arabic tokens). more information about our pre-training data is in ## argen in order to evaluate our pre-trained language models, we introduce our new benchmark for arabic language generation evaluation argen. it includes 19 different datasets with 59 test splits and covers seven tasks: machine translation (mt), codeswitched translation (cst), text summarization (ts), news title generation (ngt), question generation (qg), transliteration (tr), and paraphrasing (pph). as such, argen has wide-coverage both in terms of the number of tasks and datasets. it is also linguistically diverse as it covers both msa and various arabic dialects, in addition to arabizi (romanized arabic in the ts task) and codeswitching (in the cst task). we now describe each component of argen. ## evaluation on argen baselines and procedure. for all tasks, we compare our models to models fine-tuned with mt5 using the same training data. in addition, for mt, we compare to a vanilla sequence-to-sequence (s2s) transformer @xcite trained from scratch as implemented in fairseq @xcite . for all models and baselines, across all tasks, we identify the best model on the respective dev data and blind-test it on test data. as a rule, we report on both dev and test sets. all our dev results are in section c.2 in the appendix. ## conclusion we introduced three powerful arabic-specific textto-text transformer models trained on large msa and/or arabic dialectal data. we also introduced argen, a unified benchmark for arabic natural language generation evaluation composed of seven tasks collected from a total of 19 datasets. our models outperform mt5 on all argen tasks (52 out of 59 test sets, i.e., 88.14%). this is true even for mt involving four foreign languages from which the models have seen marginal or no pretraining data (i.e., zero-and few-shot pre-training). our models also set new sota on the large arabic language understanding evaluation benchmark arlue. our models involve vocabulary from 11 languages other than arabic, and hence can easily be further pre-trained/fine-tuned in these languages. our models are publicly available, and argen datasets are accessible from our repository. we gratefully acknowledge support from the natural sciences and engineering research council of canada (nserc; rgpin-2018-04267), the social sciences and humanities research council of canada (sshrc; 435-2018-0576; 895-2020-1004), canadian foundation for innovation (cfi; 37771), compute canada (cc), foot_15 , ubc arc-sockeye, foot_16 and advanced micro devices, inc. (amd). we thank the google tfrc program for providing us with free tpu access. foot_17 any opinions, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of nserc, sshrc, cfi, cc, arc-sockeye, amd, or google. we thank bashar talafha for help with code-switching data preparation.