Tokenizer missing? #1

by synapse - opened

I saw that some fine-tuned models got tokenizer_config.json which allows loading a processor. For instance, has got one.
Is there any reason why this is missing here?

Hello @synapse! To be honest, I don't know why this file is missing here. But as it's working without it, it seems not necessary. I think it's 'cause there is a "special_tokens_map.json" file that replaces the "tokenizer_config.json"

Sign up or log in to comment