hongzoh
/

Yi-6B_Open-Platypus-v2

Text Generation

Inference Endpoints

text-generation-inference

Model card Files Files and versions Community

Yi-6B_Open-Platypus-v2 / tokenizer_config.json

hongzoh's picture

Upload tokenizer

0c9f838 verified 3 months ago

raw history blame contribute delete

No virus

268 Bytes

	{
	"bos_token": "<\|startoftext\|>",
	"clean_up_tokenization_spaces": false,
	"eos_token": "<\|endoftext\|>",
	"legacy": true,
	"model_max_length": 4096,
	"pad_token": "<unk>",
	"sp_model_kwargs": {},
	"tokenizer_class": "LlamaTokenizer",
	"unk_token": "<unk>"
	}