Pieter Delobelle commited on
1 Parent(s): 599308b

Updated vocab.json to include unused tokens.

Browse files

The fast tokenizer complains when not all tokens from the merges.txt are
included in the vocab.json. So to fix this, the unused tokens are added
to the back.


Files changed (1) hide show
  1. vocab.json +0 -0
vocab.json CHANGED
The diff for this file is too large to render. See raw diff