cm_codeparrot / added_tokens.json
ncoop57's picture
add tokenizer
bc2ea2d
{"<eos>": 200002, "<mask>": 200000, "<bos>": 200001}