Upload README.md
Browse files
README.md
CHANGED
|
@@ -2,3 +2,5 @@
|
|
| 2 |
|
| 3 |
# summary
|
| 4 |
multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.
|
|
|
|
|
|
|
|
|
| 2 |
|
| 3 |
# summary
|
| 4 |
multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.
|
| 5 |
+
|
| 6 |
+
* vocab size: 10k
|