Create tokenizer_config.json

#2
by lysandre HF staff - opened

The model max length is currently set at a value within the transformers toolkit directly, as can be seen here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/squeezebert/tokenization_squeezebert.py

We're removing these definitions as legacy; we're therefore updating the remote checkpoints to ensure that this information is not lost. Thank you!

Closing in favor of #1

lysandre changed pull request status to closed

Sign up or log in to comment