ArabicT5: Efficient Adaptation of T5 on Arabic Language

Model Description

This model adapts T5 on the Arabic Language by pre-training T5 on :

Arabic Wikipedia.
Marefa encyclopedia.
Hindawi Books.
a collection of Arabic News.
OSCAR Dataset (32GB)

Total Corpora size is 49GB. This model uses an efficient implementation of T5 which reduces the fine-tuning and memory used Link and uses T5x for pre-training Link

Pre-training Settings and Results on TyDi QA Development Dataset ( Model in this card is highlighted in bold )

Model	Hidden Layer	Atten. head	Atten. Layers	Vocab	Hardware	Training Steps	Batch	Train x Batch Factor	Corpora
AraT5-base	768	12	12	110K	TPUv3-8	1M	128	1.0x	248GB 29B tokens (MSA + Tweets)
AraT5-msa-base	768	12	12	110K	TPUv3-8	1M	128	1.0x	70GB (MSA)
AraT5-tweets-base	768	12	12	110K	TPUv3-8	1M	128	1.0x	178GB (Tweets)
AraBART-base	768	12	12	50K	128 V100 GPUs (60h)	25 epochs	-	-	73GB (MSA)
mT5-base	768	12	12	250K	TPUv3-32	1M	1024	8.0x	6.3T tokens (mC4)
ArabicT5-17GB-small	512	8	20	32K	TPUv3-32	256K	256	0.5x	17GB (MSA)
ArabicT5-17GB-base	768	12	16	32K	TPUv3-128	500K	512	2.0x	17GB (MSA)
ArabicT5-49GB-base	768	12	16	32K	TPUv3-64	500K	256	1.0x	49GB (MSA + OSCAR)
ArabicT5-17GB-large	768	12	36	32K	TPUv3-128	500K	512	2.0x	17GB (MSA)

Results on TyDi QA, HARD, Sentiment Analysis, Sarcasm Detection ( Best Score is highlighted in bold )

Model	TyDi QA	HARD	ArSarcasm-v2-Sentiment	ArSarcasm-v2-Sarcasm	XL-SUM
AraT5-base	70.4/84.2	96.5	69.7/72.6	60.4	30.3
AraT5-msa-base	70.9/84.0	96.5	70.0/72.7	60.7	27.4
AraT5-tweets-base	65.1/79.0	96.3	70.7/73.5	61.1	25.1
mT5-base	72.2/84.1	96.2	67.3/68.8	52.2	25.7
AraBART-base	48.8/71.2	96.1	66.2/68.2	56.3	31.2
ArabicT5-17GB-small	70.8/84.8	96.4	68.9/71.2	58.9	29.2
ArabicT5-17GB-base	73.3/86.1	96.4	70.4/73.0	59.8	30.3
ArabicT5-49GB-base	73.1/85.5	96.5	71.4/74.7	60.4	30.9
ArabicT5-17GB-large	75.5/87.1	96.5	72.2/75.2	61.7	31.7

Evaluation Metrics: TyDi QA (EM/F1), HARD (Accuracy), Sentiment Analysis (Accuracy / F1-PN positive-negative), Sarcasm Detection (F1-sarcastic), XL-SUM (Rouge-L with Stemmer).

You can download the full details of our grid search for all models in all tasks above from this link: https://github.com/salrowili/ArabicT5/raw/main/ArabicT5_Grid_Search.zip

For the XL-Sum task, we choose our best run for each model using the eval set. We use the official evaluation script from XL-Sum, which uses the stemmer function, which may show better results than papers that don't use the stemmer function. The official XL-Sum paper uses a stemmer function.

In our XL-Sum results, although we show that AraT5-Base exceeded our ArabicT5-Large, in most runs, our ArabicT5-Large shows better results, as you can see from our grid search file.

Continual Pre-Training of ArabicT5 with T5x

if you want to continue pre-training ArabicT5 on your own data we have uploaded the raw t5x checkpoint to this link https://huggingface.co/sultan/ArabicT5-49GB-base/blob/main/arabict5_49GB_base_t5x.tar.gz We will share soon a tutorial on how you can do that for free with Kaggle TPU

Speedup Results

Below are our speedup results on the TyDi QA dataset, where all models have fine-tuned 13 epochs with a learning rate of 2e-4 and batch size of 3 on each device on the TPU (TPU3v-8 batch=3x8->24).

Please note these results when we fixed our hyperparameters for all models. Refer to the table above to get the best results after doing a grid search.

Model	Run Time (hh:mm:ss)	Results on TyDi QA
AraT5-msa-base	00:20:41	69.92/82.50
AraT5-base	00:20:53	68.40/81.97
AraT5-base-Tweets	00:21:17	61.67/75.96
mT5-base	00:28:24	57.98/72.81
AraBART-base	00:10:57	43.76/66.30
ArabicT5-17GB-small	00:20:00	70.79/83.85
ArabicT5-17GB-base	00:23:50	71.22/84.42
ArabicT5-17GB-large	00:52:17	72.86/86.00

Please note that we can further speed up our ArabicT5-Base by increasing the batch size since it could handle larger batch size than other base-scale models due to its hidden layer size (512).

Paper

Generative Approach for Gender-Rewriting Task with ArabicT5

FineTuning our ArabicT5 model on generative and abstractive tasks with FLAX

GitHub Page

https://github.com/salrowili/ArabicT5

Acknowledgment

We would like to acknowledge the support we have from The TPU Research Cloud (TRC) team to grant us access to TPUv3 units.

Citation

@inproceedings{alrowili-shanker-2022-generative,
    title = "Generative Approach for Gender-Rewriting Task with {A}rabic{T}5",
    author = "Alrowili, Sultan  and
      Shanker, Vijay",
    booktitle = "Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP)",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.wanlp-1.55",
    pages = "491--495",
    abstract = "Addressing the correct gender in generative tasks (e.g., Machine Translation) has been an overlooked issue in the Arabic NLP. However, the recent introduction of the Arabic Parallel Gender Corpus (APGC) dataset has established new baselines for the Arabic Gender Rewriting task. To address the Gender Rewriting task, we first pre-train our new Seq2Seq ArabicT5 model on a 17GB of Arabic Corpora. Then, we continue pre-training our ArabicT5 model on the APGC dataset using a newly proposed method. Our evaluation shows that our ArabicT5 model, when trained on the APGC dataset, achieved competitive results against existing state-of-the-art methods. In addition, our ArabicT5 model shows better results on the APGC dataset compared to other Arabic and multilingual T5 models.",
}