File size: 405 Bytes
ec2adc0 f8edff3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 |
## BERT Medium for Luxembourgish
Created from a dataset with 1M Luxembourgish sentences from Wikipedia. Corpus has approx. 16M words.
MLM objective was trained.
The BERT model has parameters `L=8` and `H=512`.
Vocabulary has 70K word pieces.
Final loss scores, after 3 epochs:
Final train loss: 4.230
Final train perplexity: 68.726
Final validation loss: 4.074
Final validation perplexity: 58.765
|