Update README.md
Browse files
README.md
CHANGED
|
@@ -13,21 +13,28 @@ tags:
|
|
| 13 |
ArabianGPT is a GPT-2 based model, custom-trained for the Arabic language, as part of the ArabianLLM initiatives at Prince Sultan University's Robotics and Internet of Things Lab.
|
| 14 |
|
| 15 |
## Key Features
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
## Training
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
## Role in ArabianLLM Initiatives
|
| 33 |
ArabianGPT 0.1B is crucial for advancing Arabic language processing, addressing challenges unique to Arabic morphology and dialects.
|
|
|
|
| 13 |
ArabianGPT is a GPT-2 based model, custom-trained for the Arabic language, as part of the ArabianLLM initiatives at Prince Sultan University's Robotics and Internet of Things Lab.
|
| 14 |
|
| 15 |
## Key Features
|
| 16 |
+
|
| 17 |
+
| Feature | Description |
|
| 18 |
+
|---------------------------|----------------------------|
|
| 19 |
+
| **Architecture** | GPT-2 |
|
| 20 |
+
| **Model Size** | 134 million parameters |
|
| 21 |
+
| **Layers** | 12 |
|
| 22 |
+
| **Model Attention Layers**| 12 (MAL) |
|
| 23 |
+
| **Context Window Size** | 768 tokens |
|
| 24 |
|
| 25 |
## Training
|
| 26 |
+
|
| 27 |
+
| Aspect | Details |
|
| 28 |
+
|---------------------|-------------------------------|
|
| 29 |
+
| **Dataset** | Abu Elkhiar Corpus |
|
| 30 |
+
| **Data Size** | 15.5 GB |
|
| 31 |
+
| **Words** | 237.8 million |
|
| 32 |
+
| **Tokens** | Over 1.75 billion |
|
| 33 |
+
| **Hardware** | NDIVIA A100 |
|
| 34 |
+
| **Training Scale** | 7.5 million examples |
|
| 35 |
+
| **Training Duration**| 3 days |
|
| 36 |
+
| **Performance** | Final loss of 3.97 |
|
| 37 |
+
|
| 38 |
|
| 39 |
## Role in ArabianLLM Initiatives
|
| 40 |
ArabianGPT 0.1B is crucial for advancing Arabic language processing, addressing challenges unique to Arabic morphology and dialects.
|