Hugging Face's logo

Spaces:
yhavinga
/
dutch-tokenizer-arena
Running

App Files Community
1
dutch-tokenizer-arena / vocab /falcon_7b /__init__.py
eson's picture
eson
add more tokenizer
293bad6
raw
history blame contribute delete
No virus
127 Bytes
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-7b", trust_remote_code=True)