zirui3
/

llm-multilingual-tokenizer

Model card Files Files and versions Community

zirui3 commited on May 24, 2023

Commit

3c6440f

·

1 Parent(s): a5864b7

Delete READE.md

Files changed (1) hide show

READE.md +0 -4

READE.md DELETED Viewed

@@ -1,4 +0,0 @@
-# summary
-multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.