Spaces:

eson
/

tokenizer-arena

Running

add more tokenizer

d2417c7 6 months ago

No virus

136 Bytes


	from transformers import AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-chat-7b", trust_remote_code=True)