Add missing merge to tokenizer

#40
by sanchit-gandhi HF staff - opened

Fixes issue 16 as described in pr 28.

sanchit-gandhi changed pull request status to merged

Sign up or log in to comment