Masked language models ( MLM) have become an integral part of many natural language processing systems. Although multilingual MLMs have been introduced to serve many languages, these have limitations as to their capacity and the size and diversity of non-English data they are pre-trained on. In this work, we remedy these issues for Arabic by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT , that have superior performance to all existing models . To evaluate our models, we propose ArBench , a new benchmark for multi-dialectal Arabic language understanding . ArBench is built using 41 datasets targeting 5 different tasks/ task clusters, allowing us to offer a series of standardized experiments under rich conditions. When fine-tuned on ArBench, ARBERT and MARBERT collectively achieve new SOTA with sizeable margins compared to all existing models such as mBERT, XLM-R (Base and Large), and AraBERT on 37 out of 45 classification taskson the 41 datasets(\%82.22) . Our models are publicly available for research .
Pre-trained language models ( LMs) are currently integral to many natural language processing systems. Although multilingual LMs were also introduced to serve many languages, these have limitations such as being costly at inference time and the size and diversity of non-English data involved in their pre-training. We remedy these issues for a collection of diverse Arabic varieties by introducing two powerful deep bidirectional transformer-based models, ARBERT and MARBERT . To evaluate our models, we also introduce ARLUE , a new benchmark for multi-dialectal Arabic language understanding evaluation. ARLUE is built using 42 datasets targeting six different task clusters, allowing us to offer a series of standardized experiments under rich conditions. When fine-tuned on ARLUE, our models collectively achieve new state-of-the-art results across the majority of tasks ( 37 out of 48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score (77.40) across all six task clusters, outperforming all other models including XLM-R Large (~ 3.4 x larger size) . Our models are publicly available at URL and ARLUE will be released through the same repository .
R
Masked
Pre-trained
R
MLM) have become an integral part of
LMs) are currently integral to
R
MLMs have been
LMs were also
R
as to their capacity
such as being costly at inference time
R
they are pre-trained on. In this work, we
involved in their pre-training. We
R
Arabic
a collection of diverse Arabic varieties
D
, that have superior performance to all existing models
R
propose ArBench
also introduce ARLUE
R
. ArBench
evaluation. ARLUE
R
41 datasets targeting 5 different tasks/
42 datasets targeting six different
R
ArBench, ARBERT and MARBERT
ARLUE, our models
R
SOTA with sizeable margins compared to all existing models such as mBERT, XLM-R (Base and Large), and AraBERT on
state-of-the-art results across the majority of tasks (
R
45 classification taskson the 41 datasets(\%82.22)
48 classification tasks, on the 42 datasets). Our best model acquires the highest ARLUE score (77.40) across all six task clusters, outperforming all other models including XLM-R Large (~ 3.4 x larger size)
R
for research
at URL and ARLUE will be released through the same repository