Create tokenizer_config.json

#1
by lysandre HF staff - opened

The model max length is currently set at a value within the transformers toolkit directly, as can be seen here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/squeezebert/tokenization_squeezebert.py

We're removing these definitions as legacy; we're therefore updating the remote checkpoints to ensure that this information is not lost. Thank you!

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment