Added model card
Browse files
README.md
CHANGED
@@ -1,12 +1,20 @@
|
|
1 |
## BERT Medium for Luxembourgish
|
2 |
|
3 |
Created from a dataset with 1M Luxembourgish sentences from Wikipedia. Corpus has approx. 16M words.
|
4 |
-
|
5 |
-
The BERT model has parameters `L=8` and `H=512`.
|
6 |
-
Vocabulary has 70K word pieces.
|
7 |
|
8 |
Final loss scores, after 3 epochs:
|
9 |
-
|
10 |
-
|
11 |
-
|
12 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
## BERT Medium for Luxembourgish
|
2 |
|
3 |
Created from a dataset with 1M Luxembourgish sentences from Wikipedia. Corpus has approx. 16M words.
|
4 |
+
|
5 |
+
The MLM objective was trained. The BERT model has parameters `L=8` and `H=512`. Vocabulary has 70K word pieces.
|
|
|
6 |
|
7 |
Final loss scores, after 3 epochs:
|
8 |
+
|
9 |
+
- Final train loss: 4.230
|
10 |
+
- Final train perplexity: 68.726
|
11 |
+
- Final validation loss: 4.074
|
12 |
+
- Final validation perplexity: 58.765
|
13 |
+
|
14 |
+
---
|
15 |
+
language:
|
16 |
+
- lu
|
17 |
+
tags:
|
18 |
+
- text
|
19 |
+
- MLM
|
20 |
+
---
|