Text Generation
Transformers
Safetensors
7 languages
stablelm
causal-lm
Inference Endpoints
12 papers
stablelm-2-1_6b / tokenizer_config.json
jon-tow's picture
fix: add default special tokens
9f2def9 verified
raw history blame
No virus
224 Bytes
{
"tokenizer_class": "Arcade100kTokenizer",
"auto_map": {
"AutoTokenizer": [
"tokenization_arcade100k.Arcade100kTokenizer",
null
]
},
"eos_token": "<|endoftext|>",
"pad_token": "<|endoftext|>"
}