Error when loading
#1
by
nicolas-dufour
- opened
When loading the tokenizer with :
transformers.CLIPTokenizer.from_pretrained("facebook/metaclip-h14-fullcc2.5b")
i get the following error:
ValueError: Non-consecutive added token '<|startoftext|>' found. Should have index 49408 but has index 49406 in saved vocabulary.
After upgrading transformers, tokenizers and huggingface-hub this issue went away for me.