Created from a dataset with 1M Luxembourgish sentences from Wikipedia. Corpus has approx. 16M words. MLM objective was trained. The BERT model has parameters L=8 and H=512. Vocabulary has 70K word pieces.
L=8
H=512
