system's picture
system HF staff
Update tokenizer_config.json
076be04
raw
history blame
411 Bytes
{"bos_token": "[SEP]", "eos_token": "[SEP]", "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "mask_token": "[MASK]", "cls_token": "[SEP]", "xprophetnet_tokenizer": true, "special_tokens_map_file": "/home/patrick/.cache/torch/transformers/556e0f8e88bd0068c08bfb686c44642a49cd93c6f73d09d48746de507490238a.39e4687b4b4265f2238315b221e397dae42a730321e6e3dbd2ef3ce2b94ac2c3", "tokenizer_file": null}