Going the other way and simplifying the tokenizer instead. 12ceb59 unverified Narsil HF staff commited on Feb 16, 2022
Revert "Making the vocab_size match the tokenizer vocab_size." 541d169 unverified Narsil HF staff commited on Feb 16, 2022
Making the vocab_size match the tokenizer vocab_size. 4be554b unverified Narsil HF staff commited on Feb 16, 2022