MuPT-v1-8192-1.97B / tokenizer_config.json
XiaoY1's picture
Upload tokenizer_config.json with huggingface_hub
4c3e062 verified
{
"model_max_length": 8192,
"tokenizer_class": "MuPTTokenizer",
"auto_map": {
"AutoTokenizer": [
"tokenization_mupt.MuPTTokenizer",
null
]
}
}