[MASK]
Arabic edition of ALBERT Base pretrained language model
The models were pretrained on ~4.4 Billion words:
Notes on training data:
albert-base | albert-large | albert-xlarge | |
---|---|---|---|
Hidden Layers | 12 | 24 | 24 |
Attention heads | 12 | 16 | 32 |
Hidden size | 768 | 1024 | 2048 |
For further details on the models performance or any other queries, please refer to Arabic-ALBERT
You can use these models by installing torch
or tensorflow
and Huggingface library transformers
. And you can use it directly by initializing it like this:
from transformers import AutoTokenizer, AutoModel
# loading the tokenizer
base_tokenizer = AutoTokenizer.from_pretrained("kuisailab/albert-base-arabic")
# loading the model
base_model = AutoModel.from_pretrained("kuisailab/albert-base-arabic")
Thanks to Google for providing free TPU for the training process and for Huggingface for hosting these models on their servers ๐