Pieter Delobelle
commited on
Commit
•
209de28
1
Parent(s):
599308b
Updated vocab.json to include unused tokens.
Browse filesThe fast tokenizer complains when not all tokens from the merges.txt are
included in the vocab.json. So to fix this, the unused tokens are added
to the back.
https://github.com/huggingface/transformers/issues/9290
- vocab.json +0 -0
vocab.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|