Upload tokenizer

#43
by ArthurZ HF staff - opened

Fixes #16, which showed that one of the merges is missing, which creates different outputs. The issue with this is that finetuned models were trained using this.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment