codegen-350M-mono / special_tokens_map.json
shpotes's picture
add tokenizer
d8bd706
raw
history blame
No virus
112 Bytes
{"bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>", "unk_token": "<|endoftext|>", "pad_token": "<pad>"}