Hugging Face's logo

Spaces:
eson
/
tokenizer-arena
Running

App Files Community
1
tokenizer-arena / vocab /aya_101 /__init__.py
eson's picture
eson
add aya
44c3329
raw
history blame
No virus
106 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("CohereForAI/aya-101")