No tokenizer works with AutoTokenizer
#1
by
etamin
- opened
When I try to use:
tokenizer = AutoTokenizer.from_pretrained("facebook/nllb-200-1.3B")
it reports:
ValueError: Tokenizer class NllbTokenizer does not exist or is not currently imported.
Seems like no NllbTokenizer here support the model nllb-200-1.3B
Hey
@etamin
, NLLB models require the NllbTokenizer, as you have seen.
These are being contributed in this PR: https://github.com/huggingface/transformers/pull/18126
Please check it out and let us know in the PR comments if it works as expected for you!
Thanks.