is this a bug?
#2
by
thehonestbob
- opened
tokenizer = FSMTTokenizer.from_pretrained(r'wmt')
s = tokenizer .encode("isn't it?")
s = tokenizer.decode(s)
print(s)
isn t it?
this bug cause by sacremoses.MosesTokenizer(lang=lang).tokenize(escape=True)