mt-dspec-legislation-en-cy / tokenizer_config.json
mgrbyte's picture
Update sources having fixed issues with tokenizer.
2bd9333 verified
raw
history blame
206 Bytes
{
"eos_token":"</s>",
"model_max_length":512,
"pad_token":"<pad>",
"separate_vocabs":false,
"source_lang":"en",
"target_lang":"cy",
"tokenizer_class":"MarianTokenizer",
"unk_token":"<unk>"
}