Spaces:

eson
/

tokenizer-arena

Running

add more tokenizers

f4973d4 7 months ago

No virus

106 Bytes


	from vocab.baichuan import tokenizer

	id1 = tokenizer.encode("<pad>")
	token1 = tokenizer.decode(125696)