Hugging Face
Models
Datasets
Spaces
Posts
Docs
Enterprise
Pricing
Log In
Sign Up
Spaces:
TeamTonic
/
Tonics-Yi-200K
like
0
Paused
App
Files
Files
Community
71dee40
Tonics-Yi-200K
/
tokenizer_config.txt
Tonic
Doing Manual Tokenizer
dc28a5a
about 1 year ago
raw
Copy download link
history
blame
Safe
201 Bytes
{
"auto_map": {
"AutoTokenizer": ["tokenization_yi.YiTokenizer", null]
},
"add_bos_token": false,
"add_eos_token": false,
"model_max_length": 200000,
"tokenizer_class": "YiTokenizer"
}