How to use the tokenizer? Is the tokenizer same as the original CLIP?
#2
by
ir1d
- opened
CLIPTokenizer.from_pretrained('laion/CLIP-ViT-B-32-laion2B-s34B-b79K') generates error
see https://github.com/mlfoundations/open_clip for usage
CLIPTokenizer.from_pretrained('laion/CLIP-ViT-B-32-laion2B-s34B-b79K')generates error
Use:
tokenizer = CLIPProcessor.from_pretrained('laion/CLIP-ViT-B-32-laion2B-s34B-b79K').tokenizer
I actually find the encoded input_ids is the same as "openai/clip-vit-base-patch32", and both tokenizers has same vocal. But there's no way to ensure.