eson's picture
add more tokenizer
c75633b
raw history blame
No virus
103 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('google/byt5-small')