roberta-base-32k / tokenizer_config.json
bab2min's picture
Update tokenizer_config.json
860ddb8 verified
{
"model_max_length": 512,
"tokenizer_class": "KiwiTokenizer",
"added_tokens_decoder": {}
}