Latest Tokenizer special character problem
#2
by
sean-lamont
- opened
I've noticed a problem when working with the latest tokenizer. It seems that the AutoTokenizer isn't correctly loading special characters, which results in the behaviour as seen below:
sean-lamont
changed discussion status to
closed
To clarify, this was a problem with my version of transformers (4.33.2), seems to work fine in 4.37.2