BERT Medium for Luxembourgish
Created from a dataset with 1M Luxembourgish sentences from Wikipedia. Corpus has approx. 16M words.
MLM objective was trained.
The BERT model has parameters L=8 and H=512.
Vocabulary has 70K word pieces.
Final loss scores, after 3 epochs: Final train loss: 4.230 Final train perplexity: 68.726 Final validation loss: 4.074 Final validation perplexity: 58.765