add tokenizer-config.json
#2
by
helpmefindaname
- opened
add the missing tokenizer-config to fix https://github.com/huggingface/transformers/issues/30643
basically, the BertTokenizer doesn't have those default values anymore and therefore acts as an infinity-length tokenizer per default.
helpmefindaname
changed pull request status to
open