Upload tokenizer

#28
by ArthurZ HF staff - opened

Fixes #16, which showed that one of the merges is missing, which creates different outputs. The issue with this is that finetuned models were trained using this.

@sanchit-gandhi if you can merge this one!

Thanks for the fix @ArthurZ !

sanchit-gandhi changed pull request status to merged

Sign up or log in to comment