Update README
#6
by
pcuenq
HF staff
- opened
No description provided.
Is AutoTokenizer / GemmaTokenizerFast no longer bugged?
The examples shown in the model card work fine in my system. Nevertheless, we should merge this PR to ensure all user-defined tokens are treated the same by both the fast and slow tokenizers.
I was wrong, the tokenizer to be fixed with https://huggingface.co/gg-hf/codegemma-2b-tokenizer.
Using GemmaTokenizer for safety. Thanks a lot
@postmasters
for noticing!
pcuenq
changed pull request status to
closed