ALUE : A rabic Language Understanding Evaluation (2021)

Abstract: The emergence of Multi-task learning (MTL)models in recent years has helped push thestate of the art in Natural Language Un-derstanding (NLU). We strongly believe thatmany NLU problems in Arabic are especiallypoised to reap the benefits of such models. Tothis end we propose the Arabic Language Un-derstanding Evaluation Benchmark (ALUE),based on 8 carefully selected and previouslypublished tasks. For five of these, we providenew privately held evaluation datasets to en-sure the fairness and validity of our benchmark. We also provide a diagnostic dataset to helpresearchers probe the inner workings of theirmodels.Our initial experiments show thatMTL models outperform their singly trainedcounterparts on most tasks. But in order to en-tice participation from the wider community,we stick to publishing singly trained baselinesonly. Nonetheless, our analysis reveals thatthere is plenty of room for improvement inArabic NLU. We hope that ALUE will playa part in helping our community realize someof these improvements. Interested researchersare invited to submit their results to our online,and publicly accessible leaderboard.

## introduction historically, research into the wide spectrum of problems in natural language processing (nlp) and understanding (nlu), has been highly compartmentalized, with each line of research attempting to tackle every single problem on its own, irrespective of the rest. however in recent years, the view has been shifting towards re-examining the whole field of nlp under a multitasking lens. this has manifested itself in the development of multi-task learning (mtl) models, which are trained to optimize multiple losses, each for a different task, simultaneously. this shift in paradigm was brought about by a confluence of various elements from the wide landscape of nlp research. for one, most core nlp tasks have been researched extensively with a significant slowdown in improvements under the banner of "singlism"! another is the recent advances in contextual word embeddings, which was brought about in turn by the advent of a whole new class of neural network architectures, namely, the transformer, as described in the seminal paper of @xcite . we believe that the community of arabic nlp is particularly poised to reap significant benefits from adopting this shift in paradigm. to this end, we seek to present a collection of 8 different arabicspecific tasks, as part of a collective benchmark which we refer to as the arabic language understanding evaluation benchmark (alue). similar to the general language understanding evaluation (glue) benchmark @xcite , the tasks we present in alue, are already available online and have featured in previous competitive workshops. to ensure fair use of the benchmark, we provide our privately held evaluation datasets for five of these tasks, in which we follow the respective original authors' annotation processes to a tee. in addition, we present an expert-constructed diagnostic dataset, to help researchers probe the inner workings of their models, and diagnose their shortcomings against a wide range of linguistic phenomena. we also present an automated and publicly published leaderboard on the internet foot_0 , open to any researcher to directly submit the results of their models to. last but not least, we include baseline results for several publicly available pre-trained arabic models. this paper aims to introduce our work to the arabic nlp community. we hope it will provide an impetus that aids the research and development ef-forts related to arabic mtl approaches, and leads to wider collaboration as well as healthy competition. in section 2, we discuss related work, both from the point of view of mtl models and datasets. in section 3, we discuss the tasks comprising the alue benchmark, and their respective datasets. section 4 focuses on the diagnostic dataset, and the way it was constructed, and the rationale behind it. an overview of our selected baselines can be found in section 5. results and discussions of our work can be found under section 6, followed by our conclusions in section 7. ## related work the idea of mtl is relatively new in nlp. one of the earliest oft-cited works in this regard was by @xcite , in which they trained a multi-layered neural model on different sentence tagging tasks, with a common sentence encoder shared between them all, achieving solid performance on all of the tasks in the process. the shift in paradigm towards mtl, requires a shift in terms of benchmarking as well. the general language understanding evaluation @xcite is one of the most widely used benchmarks for comparing mtl models in the english language. it consists of nine tasks that are based on previously published benchmarks/datasets, mainly focusing on nlu. a subsequent iteration of the benchmark, named superglue @xcite , extends the scope of focus of the original benchmark to more challenging tasks, including question answering and co-reference resolution, while including a human performance baseline. for both benchmarks, the organizers provide an automated leaderboard that serves to compare and showcase the latest advancements in the field. one of the earliest successful employment of mtl in arabic nlp was by @xcite . by using mtl in an adversarial learning setting, they reported state of the art results in crossdialectal morphological tagging. this was mainly achieved by learning dialect invariant features between msa (high resource dialect), and egyptian arabic (low resource dialect). this, they argue, helps in knowledge-transfer from the former to the latter, thereby sidestepping the issue of resource scarcity that plagues many arabic variants. they also note that the gain from such knowledge transfer approach is more significant the smaller the datasets are. another paper by @xcite employs the idea of mtl in the context of neural machine translation. for translation from dialectical arabic to modern standard arabic (msa), the authors use a sequence-to-sequence architecture, where each source language has its own encoder. however, for the target language, only a single shared decoder is used. using this setup they report better results. using this setup, they report better results and are able to efficiently use smaller datasets. freihat et al. ( 2018 ) manually curated an arabic corpus of 2 million words, that was simultaneously annotated for pos, ner, and segmentation. using an mtl model trained to perform the three aforementioned tasks, the authors were able to achieve state of the art results on said tasks, and show that such a model can greatly simplify and enhance downstream tasks, such as lemmatization. using this setup, they report better results and are able to efficiently use smaller datasets. it is worth mentioning that some of these tasks were subtasks in their respective workshops, such as the emotion classification and sentiment intensity regression subtasks from semeval-2018 task 1, and the offensive and hate speech subtasks from osact4 shared task on offensive language detection. however, for the purposes of alue, these will be treated as independent tasks rather than subtasks. nevertheless, in the discussion below, we are going to list them under the name of the original workshop task they featured in first. ## diagnostics understanding a sentence depends on the capacity of the model to capture multiple underlying linguistic representations; changes in the logical, semantic, syntactic, and/or morphological features of a sentence can alter its meaning. these alterations can be described in terms of logical relationships @xcite . the single-task training benchmark by @xcite shows varying weaknesses between different models when it comes to capturing the linguistic phenomena involved. for example, double negation is especially difficult for contextualized word vectors such as cove @xcite . nonetheless, this seems to be somewhat ameliorated by deep learning-based contextualized word representations such as elmo @xcite ). yet, elmo seems to struggle with morphological negation when compared to cove. the use of a manually annotated evaluation dataset that captures the diversity of such linguistic phenomena can help model designers better understand their models' generalization behavior, and work on improving them in the process. for this purpose, we create the alue diagnostic dataset; a high-quality, hand-labeled evaluation dataset, inspired by its glue counterpart @xcite . it is composed of around 1,100 arabic pairs of hypothesis and premise sentences. each of these is labeled with their respective entailment relationship (entailment, contradiction, or neutral) and tagged with one or more category representing the host of linguistic phenomena they represent. for the purposes of the evaluation of a given model's performance on the diagnostic dataset, we opted to follow glue's lead, and use the r3 metric @xcite , for similar reasons (i.e. unbalanced class distribution in the diagnostic dataset labels). we also believe that the r3 readily helps with the investigation of systematic errors. ## baselines all of our baselines build on publicly released pretrained word embeddings or models. these were carefully selected to represent a temporal crosssection of progress in arabic nlp during the past few years; we start with fixed word-embeddings (i.e. aravec, @xcite , and end with masked language models (i.e. arabic-bert, @xcite , by way of sentence representations (i.e. large multilingual universal sentence encoder, @xcite and the early starts of contextual embeddings (i.e. elmoformanylangs, @xcite . for bert based models, we use huggingface @xcite for implementation, whereas for all other models, we use tensorflow 2.0 @xcite . however since the elmoformany-langs model is implemented in pytorch, we decided to use its contextual embeddings without finetuning, as otherwise, we would have needed to implement it using pytorch @xcite . this would have effectively required from us to use three different deep-learning frameworks to implement all of our baselines. this, we reckon, would make the process of reproduction of our baselines much more complex for other researchers; as a matter of fact, as we trawled through the literature in search for arabic pre-trained and publicly re-leased models, we couldn't but notice the dearth of publicly released arabic models especially those that capitalize on the latest advancements in nlp. we hope that our contributions through alue will help in this regard, by being a part of a conducive environment for the arabic nlp community, to develop and publicly release state of the art mtl models. we also note the lack of large corpora dedicated in full to arabic and its varients. the majority of our selected baselines are pre-trained on dumps of arabic wikipedia and the common crawl (both of which are predominantly msa in nature), while a few of them are trained on unreleased crawls with larger coverage of the various dialects of arabic (i.e. arabicbert and aravec). this makes our baselines a little harder to compare directly, but we hope that this highlights the peculiar challenges that arabic nlp faces in this regard. for all of our models, we embed our sentences into fixed-size vectors which in turn are fed into a feed-forward network that produces the final prediction. for the universal sentence encoder (use), this is readily achieved. however, for our aravec, fasttext and elmo based models, in order to achieve the same step, we first consume the word embeddings using a bilstm. for the bert based models, this is achieved somewhat indirectly, via the [cls] token, which serves as a surrogate for sentence embeddings. more details on the exact architecture for each model per task, can be obtained via appendix a and the github repo where we release all our code foot_4 . the parameters used for each model might differ slightly, as we attempted to bring out the best performance possible from each to make our comparisons between them a little more fair. ## results and discussion several key points in our work are worth analyzing, namely: (i) the baseline scores and the comparison of the different approaches, (ii) the analysis of the performance of our baseline models on the diagnostic dataset, and (iii) the comparative analysis between private and public evaluation datasets. this section goes through each of these key points in the same order as presented above. ## conclusion in this paper, we introduce the alue benchmark, with the purpose of providing a platform for researchers interested in pushing the state of the art in arabic nlu. it consists of 8 previously published tasks, with 5 privately curated evaluation datasets to ensure the validity of the leaderboard. we evaluated the correctness of these 5 evaluation sets, finding a positive correlation between the ones we developed and the original ones. in addition, we built a novel diagnostic dataset that helps analyze the results of models against a comprehensive range of linguistic phenomena. our initial experiments show that mtl approaches outperform their single-model-per-task counterparts, but to keep our leaderboard lucrative for participation, we decided to only use single-task models as our baselines. our bert baselines seem to outperform all other models, and especially so, when the pertraining data is not confined to msa-dominant corpuses, but contain dialectical varieties of arabic as well. for the diagnostic dataset, we found that our baselines struggle to capture many of the linguistic phe-nomena represented by the dataset itself, which suggests that there is plenty of room for improvement in the state of art for arabic nlu. we hope that alue will be an integral part of the efforts to push said state of the art in the coming few years.