tiny-random-fsmt / tokenizer_config.json
LysandreJik's picture
v2 update
8ffcd27
{"langs": ["ru", "en"], "do_lower_case": false, "unk_token": "<unk>", "bos_token": "<s>", "sep_token": "</s>", "pad_token": "<pad>", "model_max_length": 1024, "special_tokens_map_file": null, "tokenizer_file": null, "name_or_path": "facebook/wmt19-ru-en", "tokenizer_class": "FSMTTokenizer"}