sdadas

/

mmlw-roberta-large

Like 16

It seems that tokenization is broken when transformers version is >= 5.6.0. Specifically, spaces are not included in tokens, which results in worse model performance.

Minimal reproducible example:

from sentence_transformers import SentenceTransformer
text = 'Hello world'
tokenizer = model.tokenizer
print(tokenizer.convert_ids_to_tokens(model.preprocess([text])['input_ids'][0]))
print(model.encode(text))

Output with transformers < 5.6.0:

['<s>', 'Hell', 'o', '▁world', '</s>']
[ 0.26690593  0.20914526  0.187053   ... -0.09326418 -0.18774515  0.01717798]

Output with transformers >= 5.6.0:

['<s>', 'Hell', 'o', 'world', '</s>']
[ 0.21954247  0.2625761  -0.01277861 ... -0.23971964 -0.08151934  0.0028541 ]

I believe that issue is present also in other models from the mmlw-roberta family.

Partial workaround includes initializing the model with additional tokenizer kwarg:

model = SentenceTransformer('sdadas/mmlw-roberta-large', processor_kwargs={'add_prefix_space': True})

But it's not ideal, as it adds a prefix space to the first token as well, so result is as follows:

['<s>', '▁Hell', 'o', '▁world', '</s>']
[ 0.27185905  0.1970694   0.1774851  ... -0.08616263 -0.16689579
  0.01360258]

It seems that issue is caused by different tokenizer class – with transformers < 5.6.0 model.tokenizer is TokenizersBackend, while with transformers >= 5.6.0 model.tokenizer is XLMRobertaTokenizer.

I believe that according to config, model should be fully compatible with XLMRobertaTokenizer class, hence I guessed this is a model issue not a transformers one.

Would be grateful for any response regarding this issue.