xu-song's picture
add more tokenizer
d2417c7
raw
history blame
136 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("internlm/internlm2-chat-7b", trust_remote_code=True)