New tokenizer.json doesn't match tokenizer.model.v3, there's no [SUFFIX], [MIDDLE] and [PREFIX] tokens.
#24
by
mukel
- opened
It seems that tokenizer.json was generated using the older tokenizer.model.v3 without the FIM tokens.
Here are a few repos that have the fix: