StripedHyena-Hessian-7B / tokenizer_config.json

Update tokenizer class in tokenizer_config,json to llama tokenizer.

f2b1452 12 months ago

149 Bytes

	{
	"unk_token": "</s>",
	"bos_token": "<s>",
	"eos_token": "</s>",
	"add_prefix_space": false,
	"tokenizer_class": "LlamaTokenizer"
	}