zirui3
/

llm-multilingual-tokenizer

Model card Files Files and versions Community

llm-multilingual-tokenizer / READE.md

zirui3's picture

Upload READE.md

a5864b7 almost 2 years ago

|

123 Bytes



	# summary
	multilingual tokenizer trained on multilingual data by using the SentencePiece library and the BPE algorithm.