loubnabnl
/

tokenizer-pile

Model card Files Files and versions Community

tokenizer-pile / special_tokens_map.json

loubnabnl's picture

loubnabnl HF staff

Upload tokenizer

694db92 12 months ago

raw history blame contribute delete

No virus

157 Bytes

	{
	"additional_special_tokens": [
	"<\|endoftext\|>"
	],
	"bos_token": "<\|endoftext\|>",
	"eos_token": "<\|endoftext\|>",
	"unk_token": "<\|endoftext\|>"
	}