[AUTO] CVST Tokenizer Badger
#49
by
pandora-s
- opened
A scripted PR to update the status of the transformer tokenizer.
> [!IMPORTANT]
> ❗
> We recommend using `mistral_common` for tokenization as the transformers tokenizer has not been tested by the Mistral team and might give incorrect results.
patrickvonplaten
changed pull request status to
merged