Opus-mt fast tokenizer
#1
by
chriszz1
- opened
Hi supermy,
I noted that you are using opus-mt model with a fast tokenizer and could you please provide some information on that. The Original opus-mt model only support slow tokenizer containing multiple files and the fast tokenizer only need 'tokenizer.json'.
您好,请问下您是如何将opus-mt的tokenizer转换成fast tokenizer,使用一个tokenizer.json文件。非常想知道如何操作,如能告知非常感谢!