|
--- |
|
language: |
|
- es |
|
tags: |
|
- albert |
|
- spanish |
|
- OpenCENIA |
|
datasets: |
|
- large_spanish_corpus |
|
--- |
|
|
|
# ALBERT Large Spanish |
|
|
|
This is an [ALBERT](https://github.com/google-research/albert) model trained on a [big spanish corpora](https://github.com/josecannete/spanish-corpora). |
|
The model was trained on a single TPU v3-8 with the following hyperparameters and steps/time: |
|
- LR: 0.000625 |
|
- Batch Size: 512 |
|
- Warmup ratio: 0.003125 |
|
- Warmup steps: 12500 |
|
- Goal steps: 4000000 |
|
- Total steps: 1450000 |
|
- Total training time (aprox): 42 days. |
|
|
|
## Training loss |
|
|
|
 |
|
|
