Hugging Face's logo

zirui3
/
llm-multilingual-tokenizer

Model card Files Community
zirui3 commited on
Commit
3315142
•
1 Parent(s): 492f8a3

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -3,4 +3,4 @@
3
  # summary
4
  multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.
5
 
6
- * vocab size: 10k
 
3
  # summary
4
  multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.
5
 
6
+ * vocab size: 100k