QARiB: QCRI Arabic and Dialectal BERT
About QARiB
QCRI Arabic and Dialectal BERT (QARiB) model, was trained on a collection of ~ 420 Million tweets and ~ 180 Million sentences of text.
For tweets, the data was collected using twitter API and using language filter. lang:ar
. For text data, it was a combination from
Arabic GigaWord, Abulkhair Arabic Corpus and OPUS.
bert-base-qarib60_860k
- Data size: 60Gb
- Number of Iterations: 860k
- Loss: 2.2454472
Training QARiB
The training of the model has been performed using Googleโs original Tensorflow code on Google Cloud TPU v2. We used a Google Cloud Storage bucket, for persistent storage of training data and models. See more details in Training QARiB
Using QARiB
You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to be fine-tuned on a downstream task. See the model hub to look for fine-tuned versions on a task that interests you. For more details, see Using QARiB
How to use
You can use this model directly with a pipeline for masked language modeling:
>>>from transformers import pipeline
>>>fill_mask = pipeline("fill-mask", model="./models/data60gb_86k")
>>> fill_mask("ุดู ุนูุฏูู
ูุง [MASK]")
[{'sequence': '[CLS] ุดู ุนูุฏูู
ูุง ุนุฑุจ [SEP]', 'score': 0.0990147516131401, 'token': 2355, 'token_str': 'ุนุฑุจ'},
{'sequence': '[CLS] ุดู ุนูุฏูู
ูุง ุฌู
ุงุนุฉ [SEP]', 'score': 0.051633741706609726, 'token': 2308, 'token_str': 'ุฌู
ุงุนุฉ'},
{'sequence': '[CLS] ุดู ุนูุฏูู
ูุง ุดุจุงุจ [SEP]', 'score': 0.046871256083250046, 'token': 939, 'token_str': 'ุดุจุงุจ'},
{'sequence': '[CLS] ุดู ุนูุฏูู
ูุง ุฑูุงู [SEP]', 'score': 0.03598872944712639, 'token': 7664, 'token_str': 'ุฑูุงู'},
{'sequence': '[CLS] ุดู ุนูุฏูู
ูุง ูุงุณ [SEP]', 'score': 0.031996358186006546, 'token': 271, 'token_str': 'ูุงุณ'}]
>>> fill_mask("ูููู ูุดููููู ูุฑุญู
[MASK]")
[{'sequence': '[CLS] ูููู ูุดููููู ูุฑุญู
ูุงูุฏูู [SEP]', 'score': 0.4152909517288208, 'token': 9650, 'token_str': 'ูุงูุฏูู'},
{'sequence': '[CLS] ูููู ูุดููููู ูุฑุญู
ูู [SEP]', 'score': 0.07663793861865997, 'token': 294, 'token_str': '##ูู'},
{'sequence': '[CLS] ูููู ูุดููููู ูุฑุญู
ุญุงูู [SEP]', 'score': 0.0453166700899601, 'token': 2663, 'token_str': 'ุญุงูู'},
{'sequence': '[CLS] ูููู ูุดููููู ูุฑุญู
ุงู
ู [SEP]', 'score': 0.04390475153923035, 'token': 1942, 'token_str': 'ุงู
ู'},
{'sequence': '[CLS] ูููู ูุดููููู ูุฑุญู
ููู [SEP]', 'score': 0.027349254116415977, 'token': 3283, 'token_str': '##ููู'}]
>>> fill_mask("ููุงู
ุงูู
ุฏูุฑ [MASK]")
[
{'sequence': '[CLS] ููุงู
ุงูู
ุฏูุฑ ุจุงูุนู
ู [SEP]', 'score': 0.0678194984793663, 'token': 4230, 'token_str': 'ุจุงูุนู
ู'},
{'sequence': '[CLS] ููุงู
ุงูู
ุฏูุฑ ุจุฐูู [SEP]', 'score': 0.05191086605191231, 'token': 984, 'token_str': 'ุจุฐูู'},
{'sequence': '[CLS] ููุงู
ุงูู
ุฏูุฑ ุจุงูุงุชุตุงู [SEP]', 'score': 0.045264165848493576, 'token': 26096, 'token_str': 'ุจุงูุงุชุตุงู'},
{'sequence': '[CLS] ููุงู
ุงูู
ุฏูุฑ ุจุนู
ูู [SEP]', 'score': 0.03732728958129883, 'token': 40486, 'token_str': 'ุจุนู
ูู'},
{'sequence': '[CLS] ููุงู
ุงูู
ุฏูุฑ ุจุงูุงู
ุฑ [SEP]', 'score': 0.0246378555893898, 'token': 29124, 'token_str': 'ุจุงูุงู
ุฑ'}
]
>>> fill_mask("ููุงู
ุช ุงูู
ุฏูุฑุฉ [MASK]")
[{'sequence': '[CLS] ููุงู
ุช ุงูู
ุฏูุฑุฉ ุจุฐูู [SEP]', 'score': 0.23992691934108734, 'token': 984, 'token_str': 'ุจุฐูู'},
{'sequence': '[CLS] ููุงู
ุช ุงูู
ุฏูุฑุฉ ุจุงูุงู
ุฑ [SEP]', 'score': 0.108805812895298, 'token': 29124, 'token_str': 'ุจุงูุงู
ุฑ'},
{'sequence': '[CLS] ููุงู
ุช ุงูู
ุฏูุฑุฉ ุจุงูุนู
ู [SEP]', 'score': 0.06639821827411652, 'token': 4230, 'token_str': 'ุจุงูุนู
ู'},
{'sequence': '[CLS] ููุงู
ุช ุงูู
ุฏูุฑุฉ ุจุงูุงุชุตุงู [SEP]', 'score': 0.05613093823194504, 'token': 26096, 'token_str': 'ุจุงูุงุชุตุงู'},
{'sequence': '[CLS] ููุงู
ุช ุงูู
ุฏูุฑุฉ ุงูู
ุฏูุฑุฉ [SEP]', 'score': 0.021778125315904617, 'token': 41635, 'token_str': 'ุงูู
ุฏูุฑุฉ'}]
Training procedure
The training of the model has been performed using Googleโs original Tensorflow code on eight core Google Cloud TPU v2. We used a Google Cloud Storage bucket, for persistent storage of training data and models.
Eval results
We evaluated QARiB models on five NLP downstream task:
- Sentiment Analysis
- Emotion Detection
- Named-Entity Recognition (NER)
- Offensive Language Detection
- Dialect Identification
The results obtained from QARiB models outperforms multilingual BERT/AraBERT/ArabicBERT.
Model Weights and Vocab Download
From Huggingface site: https://huggingface.co/qarib/bert-base-qarib60_860k
Contacts
Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish and Younes Samih
Reference
@article{abdelali2020qarib,
title={QARiB: QCRI Arabic and Dialectal BERT},
author={Ahmed, Abdelali and Sabit, Hassan and Hamdy, Mubarak and Kareem, Darwish and Younes, Samih},
link={https://github.com/qcri/QARIB},
year={2020}
}
- Downloads last month
- 11