Updated vocab.json to include unused tokens.

The fast tokenizer complains when not all tokens from the merges.txt are
included in the vocab.json. So to fix this, the unused tokens are added
to the back.

https://github.com/huggingface/transformers/issues/9290

Files changed (1) hide show

vocab.json +0 -0

vocab.json CHANGED Viewed

The diff for this file is too large to render. See raw diff